00000008:00010000:3.0:1789760656.112369:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:3.0:1789760656.112383:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760656.112394:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760656.112420:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:3.0:1789760656.112449:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495615232:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760656.141624:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495615488:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760656.174205:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971549 gen 2 00000008:00010000:3.0:1789760656.174297:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760656.174305:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760656.174315:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760656.174342:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760656.174380:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495615488:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760656.209813:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4233, 4233] 00000080:00200000:3.0:1789760656.209833:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760656.209840:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760656.209856:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17342464 i_blocks 34823 00000020:00200000:3.0:1789760656.209860:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760656.209871:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760656.209906:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17342464, count: 4096 00000080:00200000:3.0:1789760656.209913:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17342464, bytes: 4096 00000080:00200000:3.0:1789760656.209922:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760656.209931:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760656.209933:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17342464, 17346560) 0 0 00000080:00200000:3.0:1789760656.209937:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760656.209953:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4234, 4234] 00020000:00200000:3.0:1789760656.209961:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17342464 chunk: [17342464, 17346560) 17346560, 4096 00020000:00200000:3.0:1789760656.209965:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760656.209994:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4234, 4234] 00000008:00010000:3.0:1789760656.210125:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760656.210132:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760656.210143:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17342464, 17346560) append 0 append lockpos 0 00000080:00200000:3.0:1789760656.210158:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17342464=0x108a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760656.210259:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760656.210263:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17346560 00000020:00010000:3.0:1789760656.210270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760656.210273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760656.210276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760656.210280:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760656.210283:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760656.210354:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760656.210363:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760656.210379:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760656.210501:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 17342464<>17346560 00000100:00100000:3.0:1789760656.210544:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760656.210561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495615744:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760656.251554:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971551 gen 2 00000008:00010000:3.0:1789760656.251669:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:3.0:1789760656.251678:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760656.251688:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760656.251712:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:3.0:1789760656.251740:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495615744:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760656.268461:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2358:1876699495616000:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760656.268646:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495616256:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760656.271975:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2358:1876699495616000:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760656.272005:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495616256:192.168.203.109@tcp:400:kworker.0 00000008:00010000:3.0:1789760656.287703:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760656.287721:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760656.287732:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760656.287758:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760656.287791:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495615872:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760656.455210:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760656.455442:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495616896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760656.531699:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971557 gen 2 00000008:00010000:3.0:1789760656.531751:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760656.531760:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760656.531771:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760656.531799:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760656.531844:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495616896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.319291:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495621120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.347165:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971590 gen 2 00000008:00010000:3.0:1789760658.347440:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760658.347448:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760658.347460:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760658.347489:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760658.347531:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495621120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.354063:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495621248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.380861:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971591 gen 2 00000008:00010000:3.0:1789760658.380914:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760658.380924:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760658.380936:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760658.380961:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760658.380990:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495621248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.382001:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495621376:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.412096:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971592 gen 2 00000008:00010000:3.0:1789760658.412138:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760658.412144:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760658.412152:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760658.412208:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760658.412242:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495621376:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.414006:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495621504:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760658.468568:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760658.468584:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760658.468597:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760658.468622:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760658.468651:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495621504:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760658.547400:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4277, 4277] 00000080:00200000:3.0:1789760658.547430:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760658.547437:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760658.547457:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17522688 i_blocks 34823 00000020:00200000:3.0:1789760658.547463:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760658.547481:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760658.547518:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17522688, count: 4096 00000080:00200000:3.0:1789760658.547527:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17522688, bytes: 4096 00000080:00200000:3.0:1789760658.547541:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760658.547551:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760658.547553:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17522688, 17526784) 0 0 00000080:00200000:3.0:1789760658.547558:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760658.547576:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4278, 4278] 00020000:00200000:3.0:1789760658.547585:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17522688 chunk: [17522688, 17526784) 17526784, 4096 00020000:00200000:3.0:1789760658.547591:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760658.547612:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4278, 4278] 00000008:00010000:3.0:1789760658.547631:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760658.547638:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760658.547650:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17522688, 17526784) append 0 append lockpos 0 00000080:00200000:3.0:1789760658.547665:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17522688=0x10b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760658.547734:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760658.547737:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17526784 00000020:00010000:3.0:1789760658.547745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760658.547749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760658.547752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760658.547756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760658.547759:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760658.547767:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760658.547774:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760658.547817:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760658.547951:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 17522688<>17526784 00000100:00100000:3.0:1789760658.548010:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760658.548035:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495621760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.615708:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971595 gen 2 00000008:00010000:3.0:1789760658.615779:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:3.0:1789760658.615788:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760658.615798:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760658.615826:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:3.0:1789760658.615856:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495621760:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760658.654087:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4279, 4279] 00000080:00200000:3.0:1789760658.654105:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760658.654112:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760658.654128:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17530880 i_blocks 34823 00000020:00200000:3.0:1789760658.654133:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760658.654142:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760658.654240:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17530880, count: 4096 00000080:00200000:3.0:1789760658.654250:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17530880, bytes: 4096 00000080:00200000:3.0:1789760658.654261:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760658.654270:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760658.654272:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17530880, 17534976) 0 0 00000080:00200000:3.0:1789760658.654277:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760658.654294:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4280, 4280] 00020000:00200000:3.0:1789760658.654303:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17530880 chunk: [17530880, 17534976) 17534976, 4096 00020000:00200000:3.0:1789760658.654377:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760658.654458:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4280, 4280] 00000008:00010000:3.0:1789760658.654476:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760658.654482:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760658.654495:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17530880, 17534976) append 0 append lockpos 0 00000080:00200000:3.0:1789760658.654509:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17530880=0x10b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760658.654635:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760658.654639:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17534976 00000020:00010000:3.0:1789760658.654649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760658.654653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760658.654656:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760658.654659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760658.654663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760658.654671:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760658.654679:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760658.755299:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4280, 4280] 00000080:00200000:3.0:1789760658.755320:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760658.755326:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760658.757508:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495622144:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760658.794727:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971598 gen 2 00000008:00010000:3.0:1789760658.794763:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760658.794770:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760658.794779:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760658.794802:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760658.794825:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495622144:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760659.222734:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4285, 4285] 00000080:00200000:3.0:1789760659.222760:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760659.222766:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760659.222781:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17555456 i_blocks 34823 00000020:00200000:3.0:1789760659.222785:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760659.222794:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760659.222860:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17555456, count: 4096 00000080:00200000:3.0:1789760659.222868:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17555456, bytes: 4096 00000080:00200000:3.0:1789760659.223005:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760659.223015:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760659.223018:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17555456, 17559552) 0 0 00000080:00200000:3.0:1789760659.223023:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760659.223041:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4286, 4286] 00020000:00200000:3.0:1789760659.223050:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17555456 chunk: [17555456, 17559552) 17559552, 4096 00020000:00200000:3.0:1789760659.223054:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760659.223073:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4286, 4286] 00000008:00010000:3.0:1789760659.223091:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760659.223096:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760659.223109:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17555456, 17559552) append 0 append lockpos 0 00000080:00200000:3.0:1789760659.223123:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17555456=0x10be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760659.223234:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760659.223237:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17559552 00000020:00010000:3.0:1789760659.223245:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760659.223249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.223253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.223258:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760659.223261:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760659.223268:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760659.223274:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760659.223292:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760659.223412:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 17555456<>17559552 00000080:00200000:3.0:1789760659.270715:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4286, 4286] 00000080:00200000:3.0:1789760659.270741:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760659.270748:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760659.270767:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17559552 i_blocks 34823 00000020:00200000:3.0:1789760659.270773:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760659.270788:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760659.270824:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17559552, count: 4096 00000080:00200000:3.0:1789760659.270833:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17559552, bytes: 4096 00000080:00200000:3.0:1789760659.270847:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760659.270856:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760659.270859:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17559552, 17563648) 0 0 00000080:00200000:3.0:1789760659.270864:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760659.270889:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4287, 4287] 00020000:00200000:3.0:1789760659.270899:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17559552 chunk: [17559552, 17563648) 17563648, 4096 00020000:00200000:3.0:1789760659.270903:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760659.270925:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4287, 4287] 00000008:00010000:3.0:1789760659.270944:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760659.270949:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760659.270962:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17559552, 17563648) append 0 append lockpos 0 00000080:00200000:3.0:1789760659.270976:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17559552=0x10bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760659.271047:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760659.271050:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17563648 00000020:00010000:3.0:1789760659.271060:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760659.271063:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.271066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.271071:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760659.271074:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760659.271083:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760659.271090:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760659.271113:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760659.271309:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 17559552<>17563648 00000080:00200000:3.0:1789760659.325515:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4287, 4287] 00000080:00200000:3.0:1789760659.325532:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760659.325539:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760659.325563:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17563648 i_blocks 34823 00000020:00200000:3.0:1789760659.325568:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760659.325578:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760659.325613:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17563648, count: 4096 00000080:00200000:3.0:1789760659.325621:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17563648, bytes: 4096 00000080:00200000:3.0:1789760659.325630:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760659.325639:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760659.325641:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17563648, 17567744) 0 0 00000080:00200000:3.0:1789760659.325645:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760659.325662:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4288, 4288] 00020000:00200000:3.0:1789760659.325671:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17563648 chunk: [17563648, 17567744) 17567744, 4096 00020000:00200000:3.0:1789760659.325675:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760659.325695:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4288, 4288] 00000008:00010000:3.0:1789760659.325723:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760659.325730:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760659.325741:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17563648, 17567744) append 0 append lockpos 0 00000080:00200000:3.0:1789760659.325755:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17563648=0x10c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760659.325817:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760659.325820:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17567744 00000020:00010000:3.0:1789760659.325829:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760659.325833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.325836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.325840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760659.325843:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760659.325851:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760659.325858:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760659.325875:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760659.325994:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 17563648<>17567744 00000080:00200000:3.0:1789760659.390452:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4288, 4288] 00000080:00200000:3.0:1789760659.390469:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760659.390475:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760659.390490:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17567744 i_blocks 34823 00000020:00200000:3.0:1789760659.390494:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760659.390503:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760659.390536:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17567744, count: 4096 00000080:00200000:3.0:1789760659.390611:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17567744, bytes: 4096 00000080:00200000:3.0:1789760659.390624:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760659.390634:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760659.390636:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17567744, 17571840) 0 0 00000080:00200000:3.0:1789760659.390641:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760659.390659:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4289, 4289] 00020000:00200000:3.0:1789760659.390667:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17567744 chunk: [17567744, 17571840) 17571840, 4096 00020000:00200000:3.0:1789760659.390671:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760659.390692:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4289, 4289] 00000008:00010000:3.0:1789760659.390710:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760659.390716:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760659.390727:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17567744, 17571840) append 0 append lockpos 0 00000080:00200000:3.0:1789760659.390742:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17567744=0x10c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760659.390798:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760659.390801:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17571840 00000020:00010000:3.0:1789760659.390809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760659.390812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.390815:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.390819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760659.390822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760659.390829:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760659.390836:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760659.390962:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760659.391089:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 17567744<>17571840 00000100:00100000:3.0:1789760659.391337:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495623168:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760659.430082:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971605 gen 2 00000100:00100000:3.0:1789760659.430134:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203144380 x1876699495623168/t4294971606(4294971606) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760675 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760659.430213:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:3.0:1789760659.430227:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760659.430237:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760659.430261:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:3.0:1789760659.430286:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495623168:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760659.486099:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4290, 4290] 00000080:00200000:3.0:1789760659.486125:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760659.486131:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760659.486146:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17575936 i_blocks 34823 00000020:00200000:3.0:1789760659.486150:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760659.486160:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760659.486254:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17575936, count: 4096 00000080:00200000:3.0:1789760659.486262:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17575936, bytes: 4096 00000080:00200000:3.0:1789760659.486275:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760659.486284:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760659.486287:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17575936, 17580032) 0 0 00000080:00200000:3.0:1789760659.486291:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760659.486308:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4291, 4291] 00020000:00200000:3.0:1789760659.486317:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17575936 chunk: [17575936, 17580032) 17580032, 4096 00020000:00200000:3.0:1789760659.486322:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760659.486344:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4291, 4291] 00000008:00010000:3.0:1789760659.486366:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760659.486372:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760659.486385:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17575936, 17580032) append 0 append lockpos 0 00000080:00200000:3.0:1789760659.486399:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17575936=0x10c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760659.486462:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760659.486464:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17580032 00000020:00010000:3.0:1789760659.486473:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760659.486477:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.486480:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760659.486484:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760659.486486:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760659.486492:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760659.486498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760659.486515:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760659.486665:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 17575936<>17580032 00000100:00100000:3.0:1789760659.605858:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495623680:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760659.648889:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971609 gen 2 00000100:00100000:3.0:1789760659.648931:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205267100 x1876699495623680/t4294971610(4294971610) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760675 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760659.648979:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760659.648986:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760659.648996:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760659.649020:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760659.649045:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495623680:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760660.013312:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760660.013332:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495624576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760660.904126:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495626496:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760660.965328:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971631 gen 2 00000100:00100000:3.0:1789760660.965370:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205267480 x1876699495626496/t4294971632(4294971632) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760676 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760660.965418:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760660.965425:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760660.965435:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760660.965453:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760660.965477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495626496:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.199706:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495627136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.239773:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971637 gen 2 00000008:00010000:3.0:1789760661.239832:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760661.239978:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760661.239989:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760661.240011:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760661.240043:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495627136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.438140:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628032:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.520820:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971641 gen 2 00000008:00010000:3.0:1789760661.520868:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760661.520876:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760661.520887:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760661.520912:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760661.520941:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628032:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.521763:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760661.521781:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628160:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.589230:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971642 gen 2 00000008:00010000:3.0:1789760661.589282:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760661.589292:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760661.589304:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760661.589329:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760661.589359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628160:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.590421:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628288:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760661.643281:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760661.643297:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760661.643309:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760661.643339:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760661.643376:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628288:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.644825:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760661.644849:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.705731:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971644 gen 2 00000008:00010000:3.0:1789760661.705796:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760661.705805:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760661.705814:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760661.705838:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760661.705868:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.706586:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760661.861944:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971647 gen 2 00000008:00010000:3.0:1789760661.861993:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760661.862235:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760661.862249:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760661.862277:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760661.862306:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495628800:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760662.163219:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4333, 4333] 00000080:00200000:3.0:1789760662.163237:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760662.163243:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760662.163260:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17752064 i_blocks 34823 00000020:00200000:3.0:1789760662.163263:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760662.163273:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760662.163307:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17752064, count: 4096 00000080:00200000:3.0:1789760662.163315:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17752064, bytes: 4096 00000080:00200000:3.0:1789760662.163327:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760662.163336:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760662.163338:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17752064, 17756160) 0 0 00000080:00200000:3.0:1789760662.163342:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760662.163359:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4334, 4334] 00020000:00200000:3.0:1789760662.163368:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17752064 chunk: [17752064, 17756160) 17756160, 4096 00020000:00200000:3.0:1789760662.163373:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760662.163391:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4334, 4334] 00000008:00010000:3.0:1789760662.163410:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760662.163416:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760662.163428:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17752064, 17756160) append 0 append lockpos 0 00000080:00200000:3.0:1789760662.163442:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17752064=0x10ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760662.163497:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760662.163500:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17756160 00000020:00010000:3.0:1789760662.163508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760662.163512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760662.163515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760662.163518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760662.163532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760662.163541:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760662.163548:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760662.163565:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760662.163694:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 17752064<>17756160 00000100:00100000:3.0:1789760662.163745:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760662.163774:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495629312:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.234589:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971651 gen 2 00000008:00010000:3.0:1789760662.234638:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760662.234647:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760662.234658:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760662.234685:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760662.234715:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495629312:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760662.365076:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4335, 4335] 00000080:00200000:3.0:1789760662.365092:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760662.365098:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760662.365115:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17760256 i_blocks 34823 00000020:00200000:3.0:1789760662.365118:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760662.365127:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760662.365161:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17760256, count: 4096 00000080:00200000:3.0:1789760662.365226:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17760256, bytes: 4096 00000080:00200000:3.0:1789760662.365239:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760662.365249:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760662.365252:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17760256, 17764352) 0 0 00000080:00200000:3.0:1789760662.365256:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760662.365273:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4336, 4336] 00020000:00200000:3.0:1789760662.365282:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17760256 chunk: [17760256, 17764352) 17764352, 4096 00020000:00200000:3.0:1789760662.365286:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760662.365307:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4336, 4336] 00000008:00010000:3.0:1789760662.365325:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760662.365330:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760662.365340:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17760256, 17764352) append 0 append lockpos 0 00000080:00200000:3.0:1789760662.365355:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17760256=0x10f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760662.365414:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760662.365417:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17764352 00000020:00010000:3.0:1789760662.365426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760662.365430:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760662.365433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760662.365436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760662.365439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760662.365447:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760662.365455:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760662.365471:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760662.365602:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 17760256<>17764352 00000080:00200000:3.0:1789760662.457258:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4336, 4336] 00000080:00200000:3.0:1789760662.457278:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760662.457285:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760662.457301:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17764352 i_blocks 34823 00000020:00200000:3.0:1789760662.457305:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760662.457316:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760662.457348:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17764352, count: 4096 00000080:00200000:3.0:1789760662.457355:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17764352, bytes: 4096 00000080:00200000:3.0:1789760662.457367:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760662.457377:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760662.457379:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17764352, 17768448) 0 0 00000080:00200000:3.0:1789760662.457384:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760662.457401:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4337, 4337] 00020000:00200000:3.0:1789760662.457410:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17764352 chunk: [17764352, 17768448) 17768448, 4096 00020000:00200000:3.0:1789760662.457414:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760662.457433:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4337, 4337] 00000008:00010000:3.0:1789760662.457450:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760662.457457:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac430, osc lock ffff8dd205052e40, flags 0x0 00000080:00200000:3.0:1789760662.457469:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17764352, 17768448) append 0 append lockpos 0 00000080:00200000:3.0:1789760662.457485:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17764352=0x10f1000, pages 1 (max 131072), lockless, parallel 00000100:00100000:3.0:1789760662.610162:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760662.610227:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495629952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.648554:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971656 gen 2 00000008:00010000:3.0:1789760662.648599:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760662.648607:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760662.648620:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760662.648645:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760662.648675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495629952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.649684:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.706975:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971657 gen 2 00000008:00010000:3.0:1789760662.707095:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760662.707103:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760662.707115:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760662.707138:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760662.707271:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.708843:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760662.708875:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.794033:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971658 gen 2 00000008:00010000:3.0:1789760662.794088:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760662.794097:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760662.794109:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760662.794133:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760662.794166:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.800547:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630336:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760662.892365:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760662.892382:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760662.892394:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760662.892418:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760662.892448:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.893309:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760662.893329:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.977590:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971659 gen 2 00000100:00100000:3.0:1789760662.977632:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495630464/t4294971660(4294971660) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760678 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760662.977682:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760662.977689:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760662.977698:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760662.977722:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760662.977748:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760662.979588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.054543:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971661 gen 2 00000008:00010000:3.0:1789760663.054609:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760663.054617:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760663.054627:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760663.054652:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760663.054684:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.055952:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760663.055973:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630720:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.163890:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971662 gen 2 00000008:00010000:3.0:1789760663.163944:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760663.163953:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760663.163965:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760663.163990:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760663.164022:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630720:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.165480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630848:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.258963:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971663 gen 2 00000008:00010000:3.0:1789760663.259036:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760663.259045:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760663.259056:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760663.259078:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760663.259110:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495630848:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.260005:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495630976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.356925:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971664 gen 2 00000008:00010000:3.0:1789760663.356977:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760663.356986:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760663.356998:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760663.357032:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760663.357062:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495630976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.364594:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495631104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.425414:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971665 gen 2 00000008:00010000:3.0:1789760663.425465:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760663.425474:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760663.425485:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760663.425520:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760663.425551:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495631104:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760663.494451:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4349, 4349] 00000080:00200000:3.0:1789760663.494468:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760663.494474:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760663.494490:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17817600 i_blocks 34823 00000020:00200000:3.0:1789760663.494494:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760663.494504:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760663.494551:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17817600, count: 4096 00000080:00200000:3.0:1789760663.494559:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17817600, bytes: 4096 00000080:00200000:3.0:1789760663.494571:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760663.494580:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760663.494582:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17817600, 17821696) 0 0 00000080:00200000:3.0:1789760663.494588:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760663.494606:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4350, 4350] 00020000:00200000:3.0:1789760663.494615:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17817600 chunk: [17817600, 17821696) 17821696, 4096 00020000:00200000:3.0:1789760663.494620:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760663.494640:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4350, 4350] 00000008:00010000:3.0:1789760663.494658:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760663.494664:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760663.494676:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17817600, 17821696) append 0 append lockpos 0 00000080:00200000:3.0:1789760663.494690:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17817600=0x10fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760663.494750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760663.494753:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17821696 00000020:00010000:3.0:1789760663.494763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760663.494766:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.494769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.494773:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760663.494777:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760663.494785:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760663.494793:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760663.494812:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760663.494936:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 17817600<>17821696 00000100:00100000:3.0:1789760663.494997:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760663.495015:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495631360:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.592612:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971667 gen 2 00000008:00010000:3.0:1789760663.592667:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:3.0:1789760663.592732:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760663.592744:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760663.592769:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:3.0:1789760663.592800:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495631360:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760663.593332:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:3.0:1789760663.593456:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 17821696<>17825792 00000100:00100000:3.0:1789760663.593505:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495631488:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.645951:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971668 gen 2 00000008:00010000:3.0:1789760663.646002:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:3.0:1789760663.646010:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760663.646021:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760663.646107:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:3.0:1789760663.646137:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495631488:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760663.734598:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4352, 4352] 00000080:00200000:3.0:1789760663.734624:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760663.734632:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760663.734649:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17829888 i_blocks 34823 00000020:00200000:3.0:1789760663.734655:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760663.734670:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760663.734706:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17829888, count: 4096 00000080:00200000:3.0:1789760663.734714:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17829888, bytes: 4096 00000080:00200000:3.0:1789760663.734729:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760663.734739:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760663.734741:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17829888, 17833984) 0 0 00000080:00200000:3.0:1789760663.734746:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760663.734763:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4353, 4353] 00020000:00200000:3.0:1789760663.734772:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17829888 chunk: [17829888, 17833984) 17833984, 4096 00020000:00200000:3.0:1789760663.734776:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760663.734798:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4353, 4353] 00000008:00010000:3.0:1789760663.734817:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760663.734824:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760663.734836:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17829888, 17833984) append 0 append lockpos 0 00000080:00200000:3.0:1789760663.734851:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17829888=0x1101000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760663.734919:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760663.734922:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17833984 00000020:00010000:3.0:1789760663.734931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760663.734934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.734937:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.734941:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814218 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760663.734944:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760663.734951:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760663.734958:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760663.734978:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760663.735109:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 17829888<>17833984 00000100:00100000:3.0:1789760663.735239:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760663.735265:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495631744:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760663.784056:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971670 gen 2 00000008:00010000:3.0:1789760663.784137:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:3.0:1789760663.784144:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760663.784240:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760663.784256:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:3.0:1789760663.784284:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495631744:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760663.832483:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4354, 4354] 00000080:00200000:3.0:1789760663.832500:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760663.832506:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760663.833621:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000080:00200000:3.0:1789760663.886533:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4355, 4355] 00000080:00200000:3.0:1789760663.886550:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760663.886555:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760663.886570:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17842176 i_blocks 36871 00000020:00200000:3.0:1789760663.886573:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760663.886583:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760663.886646:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17842176, count: 4096 00000080:00200000:3.0:1789760663.886654:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17842176, bytes: 4096 00000080:00200000:3.0:1789760663.886677:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760663.886687:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760663.886689:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17842176, 17846272) 0 0 00000080:00200000:3.0:1789760663.886694:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760663.886709:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4356, 4356] 00020000:00200000:3.0:1789760663.886718:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17842176 chunk: [17842176, 17846272) 17846272, 4096 00020000:00200000:3.0:1789760663.886722:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760663.886742:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4356, 4356] 00000008:00010000:3.0:1789760663.886759:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760663.886765:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760663.886776:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17842176, 17846272) append 0 append lockpos 0 00000080:00200000:3.0:1789760663.886790:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17842176=0x1104000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760663.886842:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760663.886845:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17846272 00000020:00010000:3.0:1789760663.886854:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760663.886857:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.886860:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.886864:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760663.886867:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760663.886875:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760663.886881:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760663.886897:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760663.887011:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 17842176<>17846272 00000080:00200000:3.0:1789760663.940285:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4356, 4356] 00000080:00200000:3.0:1789760663.940301:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760663.940308:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760663.940324:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17846272 i_blocks 36871 00000020:00200000:3.0:1789760663.940328:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760663.940338:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760663.940372:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17846272, count: 4096 00000080:00200000:3.0:1789760663.940379:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17846272, bytes: 4096 00000080:00200000:3.0:1789760663.940388:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760663.940397:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760663.940400:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17846272, 17850368) 0 0 00000080:00200000:3.0:1789760663.940404:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760663.940418:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4357, 4357] 00020000:00200000:3.0:1789760663.940428:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17846272 chunk: [17846272, 17850368) 17850368, 4096 00020000:00200000:3.0:1789760663.940433:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760663.940450:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4357, 4357] 00000008:00010000:3.0:1789760663.940467:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760663.940473:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760663.940485:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17846272, 17850368) append 0 append lockpos 0 00000080:00200000:3.0:1789760663.940499:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17846272=0x1105000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760663.940623:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760663.940626:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17850368 00000020:00010000:3.0:1789760663.940635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760663.940639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.940642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.940647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760663.940649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760663.940674:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760663.940682:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760663.940697:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760663.940814:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 17846272<>17850368 00000080:00200000:3.0:1789760663.987372:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4357, 4357] 00000080:00200000:3.0:1789760663.987388:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760663.987394:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760663.987410:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17850368 i_blocks 36871 00000020:00200000:3.0:1789760663.987413:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760663.987422:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760663.987453:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17850368, count: 4096 00000080:00200000:3.0:1789760663.987460:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17850368, bytes: 4096 00000080:00200000:3.0:1789760663.987469:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760663.987477:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760663.987479:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17850368, 17854464) 0 0 00000080:00200000:3.0:1789760663.987483:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760663.987499:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4358, 4358] 00020000:00200000:3.0:1789760663.987507:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17850368 chunk: [17850368, 17854464) 17854464, 4096 00020000:00200000:3.0:1789760663.987512:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760663.987530:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4358, 4358] 00000008:00010000:3.0:1789760663.987547:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760663.987553:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760663.987565:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17850368, 17854464) append 0 append lockpos 0 00000080:00200000:3.0:1789760663.987579:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17850368=0x1106000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760663.987628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760663.987630:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17854464 00000020:00010000:3.0:1789760663.987639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760663.987643:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.987646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760663.987650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760663.987653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760663.987660:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760663.987666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760663.987751:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760663.987935:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 17850368<>17854464 00000100:00100000:3.0:1789760664.704907:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760664.704929:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495634048:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760664.758112:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971688 gen 2 00000008:00010000:3.0:1789760664.758228:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760664.758237:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760664.758247:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760664.758274:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760664.758424:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495634048:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760664.804947:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4372, 4372] 00000080:00200000:3.0:1789760664.804959:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760664.804965:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760664.804978:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17911808 i_blocks 36871 00000020:00200000:3.0:1789760664.804981:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760664.804990:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760664.805022:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17911808, count: 4096 00000080:00200000:3.0:1789760664.805028:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17911808, bytes: 4096 00000080:00200000:3.0:1789760664.805036:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760664.805043:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760664.805045:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17911808, 17915904) 0 0 00000080:00200000:3.0:1789760664.805049:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760664.805063:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4373, 4373] 00020000:00200000:3.0:1789760664.805069:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17911808 chunk: [17911808, 17915904) 17915904, 4096 00020000:00200000:3.0:1789760664.805072:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760664.805088:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4373, 4373] 00000008:00010000:3.0:1789760664.805103:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760664.805108:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760664.805116:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17911808, 17915904) append 0 append lockpos 0 00000080:00200000:3.0:1789760664.805127:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17911808=0x1115000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760664.805213:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760664.805216:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17915904 00000020:00010000:3.0:1789760664.805223:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760664.805226:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760664.805229:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760664.805232:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760664.805235:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760664.805241:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760664.805256:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760664.805271:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760664.805366:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 17911808<>17915904 00000100:00100000:3.0:1789760664.805406:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760664.805420:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495634304:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760664.907429:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971690 gen 2 00000008:00010000:3.0:1789760664.907466:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:3.0:1789760664.907473:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760664.907482:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760664.907508:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:3.0:1789760664.907535:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495634304:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760665.013740:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4374, 4374] 00000080:00200000:3.0:1789760665.013752:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760665.013757:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760665.013768:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17920000 i_blocks 36871 00000020:00200000:3.0:1789760665.013772:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760665.013780:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760665.013804:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17920000, count: 4096 00000080:00200000:3.0:1789760665.013809:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17920000, bytes: 4096 00000080:00200000:3.0:1789760665.013818:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760665.013824:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760665.013826:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17920000, 17924096) 0 0 00000080:00200000:3.0:1789760665.013830:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760665.013843:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4375, 4375] 00020000:00200000:3.0:1789760665.013849:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17920000 chunk: [17920000, 17924096) 17924096, 4096 00020000:00200000:3.0:1789760665.013854:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760665.013868:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4375, 4375] 00000008:00010000:3.0:1789760665.013883:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760665.013887:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760665.013896:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17920000, 17924096) append 0 append lockpos 0 00000080:00200000:3.0:1789760665.013908:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17920000=0x1117000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760665.013949:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760665.013951:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17924096 00000020:00010000:3.0:1789760665.013958:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760665.013961:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760665.013963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760665.013966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760665.013968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760665.013974:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760665.013980:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760665.013994:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760665.014080:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 17920000<>17924096 00000080:00200000:3.0:1789760665.114339:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4375, 4375] 00000080:00200000:3.0:1789760665.114352:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760665.114357:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760665.114368:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17924096 i_blocks 36871 00000020:00200000:3.0:1789760665.114371:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760665.114380:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760665.114403:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17924096, count: 4096 00000080:00200000:3.0:1789760665.114409:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17924096, bytes: 4096 00000080:00200000:3.0:1789760665.114416:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760665.114422:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760665.114424:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17924096, 17928192) 0 0 00000080:00200000:3.0:1789760665.114428:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760665.114442:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4376, 4376] 00020000:00200000:3.0:1789760665.114449:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17924096 chunk: [17924096, 17928192) 17928192, 4096 00020000:00200000:3.0:1789760665.114453:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760665.114467:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4376, 4376] 00000008:00010000:3.0:1789760665.114481:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760665.114485:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760665.114494:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17924096, 17928192) append 0 append lockpos 0 00000080:00200000:3.0:1789760665.114515:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17924096=0x1118000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760665.114559:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760665.114562:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17928192 00000020:00010000:3.0:1789760665.114569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760665.114573:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760665.114576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760665.114578:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760665.114581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760665.114588:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760665.114594:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760665.114607:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760665.114696:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 17924096<>17928192 00000080:00200000:3.0:1789760665.197073:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4376, 4376] 00000080:00200000:3.0:1789760665.197097:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760665.197105:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760665.197122:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 17928192 i_blocks 36871 00000020:00200000:3.0:1789760665.197127:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760665.197140:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760665.197218:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 17928192, count: 4096 00000080:00200000:3.0:1789760665.197227:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 17928192, bytes: 4096 00000080:00200000:3.0:1789760665.197240:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760665.197249:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760665.197251:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [17928192, 17932288) 0 0 00000080:00200000:3.0:1789760665.197256:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760665.197272:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4377, 4377] 00020000:00200000:3.0:1789760665.197280:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 17928192 chunk: [17928192, 17932288) 17932288, 4096 00020000:00200000:3.0:1789760665.197284:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760665.197305:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4377, 4377] 00000008:00010000:3.0:1789760665.197322:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760665.197328:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760665.197339:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [17928192, 17932288) append 0 append lockpos 0 00000080:00200000:3.0:1789760665.197352:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=17928192=0x1119000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760665.197418:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760665.197421:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 17932288 00000020:00010000:3.0:1789760665.197429:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760665.197433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760665.197436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760665.197440:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760665.197443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760665.197451:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760665.197459:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760665.197479:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760665.197627:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 17928192<>17932288 00000100:00100000:3.0:1789760666.405001:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495637632:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.497899:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971716 gen 2 00000008:00010000:3.0:1789760666.497934:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760666.497940:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760666.497950:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760666.497969:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760666.497992:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495637632:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.498838:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760666.498863:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495637760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.541980:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971717 gen 2 00000008:00010000:3.0:1789760666.542015:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760666.542021:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760666.542030:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760666.542051:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760666.542074:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495637760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.542698:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760666.542711:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495638272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.580686:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971718 gen 2 00000008:00010000:3.0:1789760666.580724:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760666.580730:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760666.580740:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760666.580761:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760666.580785:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495638272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.581431:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760666.581447:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495638400:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.652816:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971719 gen 2 00000008:00010000:3.0:1789760666.652849:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760666.652855:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760666.652864:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760666.652884:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760666.652907:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495638400:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.653456:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760666.653470:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495638528:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760666.712425:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760666.712438:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760666.712448:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760666.712468:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760666.712501:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495638528:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.713059:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760666.713073:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495638656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.749700:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971720 gen 2 00000100:00100000:3.0:1789760666.749732:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264e00 x1876699495638656/t4294971721(4294971721) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760682 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760666.749770:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760666.749776:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760666.749786:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760666.749806:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760666.749827:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495638656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.750633:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495638784:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.795833:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971722 gen 2 00000008:00010000:3.0:1789760666.795888:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760666.795894:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760666.795902:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760666.795969:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760666.795992:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495638784:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.798859:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760666.802341:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495638912:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760666.853356:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971723 gen 2 00000008:00010000:3.0:1789760666.853396:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760666.853403:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760666.853412:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760666.853425:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760666.853446:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495638912:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760668.102605:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760668.102671:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495642112:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760668.149650:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971748 gen 2 00000008:00010000:3.0:1789760668.149686:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760668.149692:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760668.149702:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760668.149724:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760668.149751:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495642112:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760668.285756:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4433, 4433] 00000080:00200000:3.0:1789760668.285768:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.285774:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.285785:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18161664 i_blocks 36871 00000020:00200000:3.0:1789760668.285789:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.285798:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.285822:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18161664, count: 4096 00000080:00200000:3.0:1789760668.285828:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18161664, bytes: 4096 00000080:00200000:3.0:1789760668.285837:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.285844:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.285846:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18161664, 18165760) 0 0 00000080:00200000:3.0:1789760668.285864:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.285876:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4434, 4434] 00020000:00200000:3.0:1789760668.285883:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18161664 chunk: [18161664, 18165760) 18165760, 4096 00020000:00200000:3.0:1789760668.285887:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.285901:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4434, 4434] 00000008:00010000:3.0:1789760668.285914:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.285920:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.285928:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18161664, 18165760) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.285939:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18161664=0x1152000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.285984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.285987:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18165760 00000020:00010000:3.0:1789760668.285994:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.285997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.286000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.286003:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.286006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.286012:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.286017:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.286043:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760668.286138:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 18161664<>18165760 00000080:00200000:3.0:1789760668.323454:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4434, 4434] 00000080:00200000:3.0:1789760668.323467:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.323472:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.323491:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18165760 i_blocks 36871 00000020:00200000:3.0:1789760668.323495:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.323503:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.323526:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18165760, count: 4096 00000080:00200000:3.0:1789760668.323531:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18165760, bytes: 4096 00000080:00200000:3.0:1789760668.323539:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.323546:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.323548:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18165760, 18169856) 0 0 00000080:00200000:3.0:1789760668.323552:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.323564:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4435, 4435] 00020000:00200000:3.0:1789760668.323571:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18165760 chunk: [18165760, 18169856) 18169856, 4096 00020000:00200000:3.0:1789760668.323574:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.323589:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4435, 4435] 00000008:00010000:3.0:1789760668.323601:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.323607:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.323615:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18165760, 18169856) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.323626:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18165760=0x1153000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.323666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.323669:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18169856 00000020:00010000:3.0:1789760668.323676:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.323679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.323682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.323685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.323688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.323694:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.323700:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.323714:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760668.323802:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 18165760<>18169856 00000080:00200000:3.0:1789760668.370652:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4435, 4435] 00000080:00200000:3.0:1789760668.370666:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.370672:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.370685:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18169856 i_blocks 36871 00000020:00200000:3.0:1789760668.370689:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.370697:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.370719:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18169856, count: 4096 00000080:00200000:3.0:1789760668.370725:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18169856, bytes: 4096 00000080:00200000:3.0:1789760668.370733:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.370740:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.370742:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18169856, 18173952) 0 0 00000080:00200000:3.0:1789760668.370746:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.370759:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4436, 4436] 00020000:00200000:3.0:1789760668.370765:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18169856 chunk: [18169856, 18173952) 18173952, 4096 00020000:00200000:3.0:1789760668.370769:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.370783:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4436, 4436] 00000008:00010000:3.0:1789760668.370797:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.370802:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.370810:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18169856, 18173952) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.370822:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18169856=0x1154000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.370867:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.370870:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18173952 00000020:00010000:3.0:1789760668.370876:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.370880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.370882:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.370886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.370888:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.370895:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.370899:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.370911:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760668.371005:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 18169856<>18173952 00000080:00200000:3.0:1789760668.423275:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4436, 4436] 00000080:00200000:3.0:1789760668.423290:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.423295:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.423309:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18173952 i_blocks 36871 00000020:00200000:3.0:1789760668.423313:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.423322:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.423349:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18173952, count: 4096 00000080:00200000:3.0:1789760668.423355:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18173952, bytes: 4096 00000080:00200000:3.0:1789760668.423363:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.423370:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.423372:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18173952, 18178048) 0 0 00000080:00200000:3.0:1789760668.423376:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.423390:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4437, 4437] 00020000:00200000:3.0:1789760668.423434:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18173952 chunk: [18173952, 18178048) 18178048, 4096 00020000:00200000:3.0:1789760668.423439:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.423482:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4437, 4437] 00000008:00010000:3.0:1789760668.423497:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.423502:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.423512:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18173952, 18178048) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.423522:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18173952=0x1155000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.423566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.423569:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18178048 00000020:00010000:3.0:1789760668.423576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.423579:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.423582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.423585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.423587:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.423594:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.423600:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.423614:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760668.423708:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 18173952<>18178048 00000080:00200000:3.0:1789760668.488503:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4437, 4437] 00000080:00200000:3.0:1789760668.488521:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.488527:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.488542:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18178048 i_blocks 36871 00000020:00200000:3.0:1789760668.488546:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.488556:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.488591:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18178048, count: 4096 00000080:00200000:3.0:1789760668.488598:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18178048, bytes: 4096 00000080:00200000:3.0:1789760668.488612:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.488621:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.488624:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18178048, 18182144) 0 0 00000080:00200000:3.0:1789760668.488630:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.488649:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4438, 4438] 00020000:00200000:3.0:1789760668.488658:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18178048 chunk: [18178048, 18182144) 18182144, 4096 00020000:00200000:3.0:1789760668.488663:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.488682:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4438, 4438] 00000008:00010000:3.0:1789760668.488701:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.488708:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.488718:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18178048, 18182144) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.488732:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18178048=0x1156000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.488792:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.488795:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18182144 00000020:00010000:3.0:1789760668.488804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.488807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.488811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.488814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.488817:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.488823:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.488830:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.488848:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760668.488975:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 18178048<>18182144 00000080:00200000:3.0:1789760668.537164:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4438, 4438] 00000080:00200000:3.0:1789760668.537235:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.537241:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.537256:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18182144 i_blocks 36871 00000020:00200000:3.0:1789760668.537262:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.537276:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.537304:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18182144, count: 4096 00000080:00200000:3.0:1789760668.537311:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18182144, bytes: 4096 00000080:00200000:3.0:1789760668.537321:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.537329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.537331:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18182144, 18186240) 0 0 00000080:00200000:3.0:1789760668.537336:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.537350:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4439, 4439] 00020000:00200000:3.0:1789760668.537357:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18182144 chunk: [18182144, 18186240) 18186240, 4096 00020000:00200000:3.0:1789760668.537362:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.537379:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4439, 4439] 00000008:00010000:3.0:1789760668.537395:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.537401:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.537411:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18182144, 18186240) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.537424:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18182144=0x1157000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.537485:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.537488:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18186240 00000020:00010000:3.0:1789760668.537495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.537498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.537501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.537504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.537507:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.537513:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.537518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.537536:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760668.537631:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 18182144<>18186240 00000080:00200000:3.0:1789760668.574388:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4439, 4439] 00000080:00200000:3.0:1789760668.574400:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.574406:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.574418:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18186240 i_blocks 36871 00000020:00200000:3.0:1789760668.574421:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.574429:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.574452:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18186240, count: 4096 00000080:00200000:3.0:1789760668.574458:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18186240, bytes: 4096 00000080:00200000:3.0:1789760668.574466:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.574481:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.574483:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18186240, 18190336) 0 0 00000080:00200000:3.0:1789760668.574488:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.574500:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4440, 4440] 00020000:00200000:3.0:1789760668.574506:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18186240 chunk: [18186240, 18190336) 18190336, 4096 00020000:00200000:3.0:1789760668.574510:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.574535:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4440, 4440] 00000008:00010000:3.0:1789760668.574548:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.574553:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.574569:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18186240, 18190336) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.574579:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18186240=0x1158000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.574625:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.574628:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18190336 00000020:00010000:3.0:1789760668.574635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.574639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.574641:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.574644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.574647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.574653:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.574658:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.574671:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760668.574764:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 18186240<>18190336 00000080:00200000:3.0:1789760668.612399:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4440, 4440] 00000080:00200000:3.0:1789760668.612412:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.612417:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.612429:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18190336 i_blocks 36871 00000020:00200000:3.0:1789760668.612432:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.612439:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.612464:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18190336, count: 4096 00000080:00200000:3.0:1789760668.612469:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18190336, bytes: 4096 00000080:00200000:3.0:1789760668.612485:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.612492:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.612495:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18190336, 18194432) 0 0 00000080:00200000:3.0:1789760668.612499:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.612511:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4441, 4441] 00020000:00200000:3.0:1789760668.612525:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18190336 chunk: [18190336, 18194432) 18194432, 4096 00020000:00200000:3.0:1789760668.612530:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.612545:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4441, 4441] 00000008:00010000:3.0:1789760668.612557:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.612562:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.612570:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18190336, 18194432) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.612580:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18190336=0x1159000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.612622:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.612625:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18194432 00000020:00010000:3.0:1789760668.612632:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.612635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.612638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.612641:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.612644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.612650:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.612655:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.612669:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760668.612751:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 18190336<>18194432 00000080:00200000:3.0:1789760668.655359:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4441, 4441] 00000080:00200000:3.0:1789760668.655371:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.655377:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.655390:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18194432 i_blocks 36871 00000020:00200000:3.0:1789760668.655394:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.655401:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.655426:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18194432, count: 4096 00000080:00200000:3.0:1789760668.655432:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18194432, bytes: 4096 00000080:00200000:3.0:1789760668.655440:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.655447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.655450:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18194432, 18198528) 0 0 00000080:00200000:3.0:1789760668.655454:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.655467:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4442, 4442] 00020000:00200000:3.0:1789760668.655481:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18194432 chunk: [18194432, 18198528) 18198528, 4096 00020000:00200000:3.0:1789760668.655485:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.655500:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4442, 4442] 00000008:00010000:3.0:1789760668.655580:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.655585:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.655595:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18194432, 18198528) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.655606:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18194432=0x115a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.655650:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.655653:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18198528 00000020:00010000:3.0:1789760668.655659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.655663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.655665:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.655669:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.655671:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.655677:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.655682:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.655695:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760668.655784:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 18194432<>18198528 00000080:00200000:3.0:1789760668.710719:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4442, 4442] 00000080:00200000:3.0:1789760668.710735:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760668.710741:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760668.710753:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18198528 i_blocks 36871 00000020:00200000:3.0:1789760668.710757:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760668.710767:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760668.710793:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18198528, count: 4096 00000080:00200000:3.0:1789760668.710799:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18198528, bytes: 4096 00000080:00200000:3.0:1789760668.710808:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760668.710815:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760668.710817:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18198528, 18202624) 0 0 00000080:00200000:3.0:1789760668.710822:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760668.710836:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4443, 4443] 00020000:00200000:3.0:1789760668.710844:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18198528 chunk: [18198528, 18202624) 18202624, 4096 00020000:00200000:3.0:1789760668.710847:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760668.710874:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4443, 4443] 00000008:00010000:3.0:1789760668.710887:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760668.710893:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760668.710902:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18198528, 18202624) append 0 append lockpos 0 00000080:00200000:3.0:1789760668.710913:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18198528=0x115b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760668.710964:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760668.710966:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18202624 00000020:00010000:3.0:1789760668.710974:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760668.710977:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.710980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760668.710983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760668.710986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760668.710993:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760668.710997:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760668.711016:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760668.711556:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 18198528<>18202624 00000100:00100000:3.0:1789760668.922158:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760668.922218:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644160:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760668.970508:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971764 gen 2 00000008:00010000:3.0:1789760668.970558:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760668.970565:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760668.970577:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760668.970672:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760668.970698:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644160:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760668.971796:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644288:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.020980:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971765 gen 2 00000008:00010000:3.0:1789760669.021014:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760669.021021:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760669.021030:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760669.021051:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760669.021075:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644288:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.023991:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760669.024010:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644416:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760669.068781:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760669.068795:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760669.068805:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760669.068870:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760669.068895:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.070055:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.142678:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971767 gen 2 00000008:00010000:3.0:1789760669.142724:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760669.142732:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760669.142740:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760669.142804:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760669.142827:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.147525:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760669.148438:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644672:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.189995:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971768 gen 2 00000008:00010000:3.0:1789760669.190032:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760669.190039:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760669.190048:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760669.190301:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760669.190329:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495644672:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760669.248572:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4452, 4452] 00000080:00200000:3.0:1789760669.248588:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.248595:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.248607:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18239488 i_blocks 36871 00000020:00200000:3.0:1789760669.248610:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.248618:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.248645:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18239488, count: 4096 00000080:00200000:3.0:1789760669.248651:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18239488, bytes: 4096 00000080:00200000:3.0:1789760669.248659:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.248666:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.248668:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18239488, 18243584) 0 0 00000080:00200000:3.0:1789760669.248672:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.248684:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4453, 4453] 00020000:00200000:3.0:1789760669.248691:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18239488 chunk: [18239488, 18243584) 18243584, 4096 00020000:00200000:3.0:1789760669.248694:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.248710:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4453, 4453] 00000008:00010000:3.0:1789760669.248734:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.248739:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.248748:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18239488, 18243584) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.248758:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18239488=0x1165000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.248802:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.248804:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18243584 00000020:00010000:3.0:1789760669.248811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.248814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.248817:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.248820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.248823:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.248829:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.248835:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.248860:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760669.248953:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 18239488<>18243584 00000080:00200000:3.0:1789760669.297727:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4453, 4453] 00000080:00200000:3.0:1789760669.297750:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.297755:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.297770:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18243584 i_blocks 36871 00000020:00200000:3.0:1789760669.297775:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.297787:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.297815:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18243584, count: 4096 00000080:00200000:3.0:1789760669.297821:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18243584, bytes: 4096 00000080:00200000:3.0:1789760669.297832:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.297840:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.297846:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18243584, 18247680) 0 0 00000080:00200000:3.0:1789760669.297850:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.297864:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4454, 4454] 00020000:00200000:3.0:1789760669.297872:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18243584 chunk: [18243584, 18247680) 18247680, 4096 00020000:00200000:3.0:1789760669.297875:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.297893:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4454, 4454] 00000008:00010000:3.0:1789760669.297907:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.297913:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.297921:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18243584, 18247680) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.297932:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18243584=0x1166000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.297988:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.297990:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18247680 00000020:00010000:3.0:1789760669.297997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.298000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.298003:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.298006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.298008:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.298016:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.298021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.298040:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760669.298134:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 18243584<>18247680 00000080:00200000:3.0:1789760669.347052:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4454, 4454] 00000080:00200000:3.0:1789760669.347063:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.347067:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.347078:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18247680 i_blocks 36871 00000020:00200000:3.0:1789760669.347080:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.347086:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.347107:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18247680, count: 4096 00000080:00200000:3.0:1789760669.347111:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18247680, bytes: 4096 00000080:00200000:3.0:1789760669.347119:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.347124:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.347126:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18247680, 18251776) 0 0 00000080:00200000:3.0:1789760669.347129:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.347142:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4455, 4455] 00020000:00200000:3.0:1789760669.347156:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18247680 chunk: [18247680, 18251776) 18251776, 4096 00020000:00200000:3.0:1789760669.347160:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.347212:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4455, 4455] 00000008:00010000:3.0:1789760669.347226:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.347231:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.347241:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18247680, 18251776) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.347251:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18247680=0x1167000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.347294:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.347297:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18251776 00000020:00010000:3.0:1789760669.347304:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.347307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.347310:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.347313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.347316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.347323:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.347329:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.347348:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760669.347441:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 18247680<>18251776 00000080:00200000:3.0:1789760669.405709:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4455, 4455] 00000080:00200000:3.0:1789760669.405733:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.405739:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.405752:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18251776 i_blocks 36871 00000020:00200000:3.0:1789760669.405756:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.405763:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.405790:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18251776, count: 4096 00000080:00200000:3.0:1789760669.405796:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18251776, bytes: 4096 00000080:00200000:3.0:1789760669.405804:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.405811:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.405813:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18251776, 18255872) 0 0 00000080:00200000:3.0:1789760669.405817:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.405831:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4456, 4456] 00020000:00200000:3.0:1789760669.405838:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18251776 chunk: [18251776, 18255872) 18255872, 4096 00020000:00200000:3.0:1789760669.405841:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.405868:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4456, 4456] 00000008:00010000:3.0:1789760669.405882:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.405887:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.405896:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18251776, 18255872) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.405907:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18251776=0x1168000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.405953:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.405956:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18255872 00000020:00010000:3.0:1789760669.405963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.405966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.405969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.405972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.405975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.405982:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.405987:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.406002:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760669.406091:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 18251776<>18255872 00000080:00200000:3.0:1789760669.493764:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4456, 4456] 00000080:00200000:3.0:1789760669.493779:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.493784:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.493797:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18255872 i_blocks 36871 00000020:00200000:3.0:1789760669.493800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.493808:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.493835:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18255872, count: 4096 00000080:00200000:3.0:1789760669.493841:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18255872, bytes: 4096 00000080:00200000:3.0:1789760669.493850:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.493857:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.493859:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18255872, 18259968) 0 0 00000080:00200000:3.0:1789760669.493863:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.493879:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4457, 4457] 00020000:00200000:3.0:1789760669.493887:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18255872 chunk: [18255872, 18259968) 18259968, 4096 00020000:00200000:3.0:1789760669.493891:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.493910:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4457, 4457] 00000008:00010000:3.0:1789760669.493928:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.493946:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.493958:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18255872, 18259968) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.493973:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18255872=0x1169000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.494039:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.494042:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18259968 00000020:00010000:3.0:1789760669.494050:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.494053:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.494057:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.494060:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.494062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.494068:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.494074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.494089:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760669.494349:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 18255872<>18259968 00000080:00200000:3.0:1789760669.581881:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4457, 4457] 00000080:00200000:3.0:1789760669.581894:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.581899:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.581911:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18259968 i_blocks 36871 00000020:00200000:3.0:1789760669.581915:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.581924:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.581951:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18259968, count: 4096 00000080:00200000:3.0:1789760669.581957:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18259968, bytes: 4096 00000080:00200000:3.0:1789760669.581968:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.581975:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.581977:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18259968, 18264064) 0 0 00000080:00200000:3.0:1789760669.581981:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.581993:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4458, 4458] 00020000:00200000:3.0:1789760669.582000:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18259968 chunk: [18259968, 18264064) 18264064, 4096 00020000:00200000:3.0:1789760669.582004:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.582019:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4458, 4458] 00000008:00010000:3.0:1789760669.582032:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.582038:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.582047:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18259968, 18264064) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.582058:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18259968=0x116a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.582101:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.582104:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18264064 00000020:00010000:3.0:1789760669.582111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.582114:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.582118:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.582121:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.582124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.582131:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.582137:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.582163:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760669.582298:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 18259968<>18264064 00000080:00200000:3.0:1789760669.632426:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4458, 4458] 00000080:00200000:3.0:1789760669.632449:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.632457:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.632487:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18264064 i_blocks 36871 00000020:00200000:3.0:1789760669.632492:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.632506:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.632555:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18264064, count: 4096 00000080:00200000:3.0:1789760669.632563:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18264064, bytes: 4096 00000080:00200000:3.0:1789760669.632574:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.632584:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.632587:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18264064, 18268160) 0 0 00000080:00200000:3.0:1789760669.632591:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.632610:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4459, 4459] 00020000:00200000:3.0:1789760669.632619:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18264064 chunk: [18264064, 18268160) 18268160, 4096 00020000:00200000:3.0:1789760669.632624:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.632646:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4459, 4459] 00000008:00010000:3.0:1789760669.632669:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.632676:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.632690:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18264064, 18268160) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.632705:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18264064=0x116b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.632773:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.632776:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18268160 00000020:00010000:3.0:1789760669.632787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.632791:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.632795:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.632799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.632803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.632813:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.632820:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.632844:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760669.632993:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 18264064<>18268160 00000080:00200000:3.0:1789760669.674683:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4459, 4459] 00000080:00200000:3.0:1789760669.674706:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.674714:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.674731:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18268160 i_blocks 36871 00000020:00200000:3.0:1789760669.674736:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.674749:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.674786:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18268160, count: 4096 00000080:00200000:3.0:1789760669.674794:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18268160, bytes: 4096 00000080:00200000:3.0:1789760669.674809:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.674820:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.674822:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18268160, 18272256) 0 0 00000080:00200000:3.0:1789760669.674827:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.674843:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4460, 4460] 00020000:00200000:3.0:1789760669.674869:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18268160 chunk: [18268160, 18272256) 18272256, 4096 00020000:00200000:3.0:1789760669.674874:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.674896:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4460, 4460] 00000008:00010000:3.0:1789760669.674914:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.674920:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.674933:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18268160, 18272256) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.674946:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18268160=0x116c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.675015:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.675018:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18272256 00000020:00010000:3.0:1789760669.675027:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.675031:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.675034:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.675038:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.675041:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.675050:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.675056:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.675077:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760669.675568:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 18268160<>18272256 00000100:00100000:3.0:1789760669.675615:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760669.675637:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495645824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.755557:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971777 gen 2 00000008:00010000:3.0:1789760669.755617:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:3.0:1789760669.755623:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760669.755631:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760669.755661:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:3.0:1789760669.755682:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495645824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.759315:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495645952:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760669.789290:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760669.789304:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760669.789314:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760669.789336:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760669.789365:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495645952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.791227:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760669.791262:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495646080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760669.834835:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971779 gen 2 00000008:00010000:3.0:1789760669.834986:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760669.834995:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760669.835006:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760669.835028:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760669.835059:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495646080:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760669.892836:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4463, 4463] 00000080:00200000:3.0:1789760669.892848:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.892854:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.892866:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18284544 i_blocks 36871 00000020:00200000:3.0:1789760669.892870:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.892887:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.892913:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18284544, count: 4096 00000080:00200000:3.0:1789760669.892919:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18284544, bytes: 4096 00000080:00200000:3.0:1789760669.892927:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.892935:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.892936:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18284544, 18288640) 0 0 00000080:00200000:3.0:1789760669.892941:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.892954:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4464, 4464] 00020000:00200000:3.0:1789760669.892960:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18284544 chunk: [18284544, 18288640) 18288640, 4096 00020000:00200000:3.0:1789760669.892964:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.892979:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4464, 4464] 00000008:00010000:3.0:1789760669.892992:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.892997:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.893004:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18284544, 18288640) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.893014:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18284544=0x1170000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.893057:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.893059:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18288640 00000020:00010000:3.0:1789760669.893066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.893069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.893072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.893075:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.893077:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.893083:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.893088:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.893101:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760669.893342:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 18284544<>18288640 00000080:00200000:3.0:1789760669.970731:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4464, 4464] 00000080:00200000:3.0:1789760669.970745:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760669.970750:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760669.970762:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18288640 i_blocks 36871 00000020:00200000:3.0:1789760669.970765:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760669.970773:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760669.970796:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18288640, count: 4096 00000080:00200000:3.0:1789760669.970801:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18288640, bytes: 4096 00000080:00200000:3.0:1789760669.970810:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760669.970816:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760669.970819:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18288640, 18292736) 0 0 00000080:00200000:3.0:1789760669.970822:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760669.970834:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4465, 4465] 00020000:00200000:3.0:1789760669.970841:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18288640 chunk: [18288640, 18292736) 18292736, 4096 00020000:00200000:3.0:1789760669.970845:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760669.970861:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4465, 4465] 00000008:00010000:3.0:1789760669.970891:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760669.970897:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760669.970907:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18288640, 18292736) append 0 append lockpos 0 00000080:00200000:3.0:1789760669.970917:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18288640=0x1171000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760669.970961:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760669.970963:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18292736 00000020:00010000:3.0:1789760669.970970:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760669.970973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.970976:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760669.970979:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760669.970982:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760669.970988:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760669.970995:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760669.971009:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760669.971104:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 18288640<>18292736 00000100:00100000:3.0:1789760669.971594:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760669.971680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495646464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.039614:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971782 gen 2 00000008:00010000:3.0:1789760670.039654:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760670.039661:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.039671:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.039695:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760670.039717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495646464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.218977:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495646976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.269749:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971786 gen 2 00000008:00010000:3.0:1789760670.269788:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760670.269794:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.269802:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.269823:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760670.269846:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495646976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.270642:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760670.270658:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495647104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.330413:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971787 gen 2 00000008:00010000:3.0:1789760670.330453:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760670.330459:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.330468:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.330496:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760670.330520:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495647104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.331718:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495647232:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.376663:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971788 gen 2 00000008:00010000:3.0:1789760670.376696:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760670.376702:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.376710:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.376728:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760670.376750:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495647232:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.377478:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760670.377500:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495647360:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760670.422276:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760670.422289:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.422302:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.422321:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760670.422346:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495647360:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.423213:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760670.423228:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495647488:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.464424:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971790 gen 2 00000008:00010000:3.0:1789760670.464469:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760670.464475:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.464483:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.464503:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760670.464527:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495647488:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.465259:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760670.465272:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495647616:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.505001:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971791 gen 2 00000008:00010000:3.0:1789760670.505038:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760670.505045:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.505055:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.505079:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760670.505102:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495647616:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.505832:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760670.505848:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495647744:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.542873:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971792 gen 2 00000008:00010000:3.0:1789760670.542914:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760670.542922:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.542932:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.542956:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760670.542979:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495647744:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.545101:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495647872:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.588047:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971793 gen 2 00000008:00010000:3.0:1789760670.588084:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760670.588092:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.588101:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.588212:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760670.588237:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495647872:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760670.589800:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495648000:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760670.635950:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760670.635961:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760670.635970:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760670.635987:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760670.636010:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495648000:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.060021:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760671.060051:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495649280:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760671.101269:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760671.101286:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760671.101300:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760671.101324:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760671.101359:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495649280:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.104000:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760671.104029:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495649408:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.241485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495649792:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.273957:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971808 gen 2 00000008:00010000:3.0:1789760671.274008:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760671.274017:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760671.274027:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760671.274107:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760671.274156:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495649792:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.276275:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495649920:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.324636:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971809 gen 2 00000008:00010000:3.0:1789760671.324771:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760671.324780:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760671.324794:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760671.324892:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760671.324925:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495649920:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.327015:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760671.327106:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495650048:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.376543:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971810 gen 2 00000008:00010000:3.0:1789760671.376595:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760671.376603:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760671.376615:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760671.376639:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760671.376697:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495650048:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760671.460406:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4495, 4495] 00000080:00200000:3.0:1789760671.460423:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760671.460429:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760671.460446:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18415616 i_blocks 36871 00000020:00200000:3.0:1789760671.460459:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760671.460469:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760671.460504:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18415616, count: 4096 00000080:00200000:3.0:1789760671.460510:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18415616, bytes: 4096 00000080:00200000:3.0:1789760671.460520:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760671.460529:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760671.460532:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18415616, 18419712) 0 0 00000080:00200000:3.0:1789760671.460536:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760671.460553:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4496, 4496] 00020000:00200000:3.0:1789760671.460562:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18415616 chunk: [18415616, 18419712) 18419712, 4096 00020000:00200000:3.0:1789760671.460567:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760671.460587:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4496, 4496] 00000008:00010000:3.0:1789760671.460605:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760671.460611:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760671.460623:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18415616, 18419712) append 0 append lockpos 0 00000080:00200000:3.0:1789760671.460638:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18415616=0x1190000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760671.460693:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760671.460696:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18419712 00000020:00010000:3.0:1789760671.460704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760671.460707:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.460710:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.460713:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760671.460716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760671.460723:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760671.460731:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760671.460747:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760671.460887:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 18415616<>18419712 00000100:00100000:3.0:1789760671.460927:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760671.460943:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495650432:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760671.498475:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760671.498491:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760671.498504:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760671.498526:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760671.498558:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495650432:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760671.533489:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4497, 4497] 00000080:00200000:3.0:1789760671.533506:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760671.533513:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760671.533528:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18423808 i_blocks 36871 00000020:00200000:3.0:1789760671.533532:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760671.533541:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760671.533578:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18423808, count: 4096 00000080:00200000:3.0:1789760671.533585:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18423808, bytes: 4096 00000080:00200000:3.0:1789760671.533595:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760671.533604:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760671.533607:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18423808, 18427904) 0 0 00000080:00200000:3.0:1789760671.533611:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760671.533629:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4498, 4498] 00020000:00200000:3.0:1789760671.533637:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18423808 chunk: [18423808, 18427904) 18427904, 4096 00020000:00200000:3.0:1789760671.533642:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760671.533662:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4498, 4498] 00000008:00010000:3.0:1789760671.533680:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760671.533686:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760671.533697:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18423808, 18427904) append 0 append lockpos 0 00000080:00200000:3.0:1789760671.533711:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18423808=0x1192000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760671.533771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760671.533773:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18427904 00000020:00010000:3.0:1789760671.533782:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760671.533785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.533788:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.533791:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760671.533794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760671.533801:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760671.533808:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760671.533824:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760671.533955:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 18423808<>18427904 00000080:00200000:3.0:1789760671.579767:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4498, 4498] 00000080:00200000:3.0:1789760671.579786:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760671.579793:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760671.579810:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18427904 i_blocks 36871 00000020:00200000:3.0:1789760671.579814:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760671.579825:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760671.579954:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18427904, count: 4096 00000080:00200000:3.0:1789760671.579963:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18427904, bytes: 4096 00000080:00200000:3.0:1789760671.579974:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760671.579984:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760671.579986:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18427904, 18432000) 0 0 00000080:00200000:3.0:1789760671.579991:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760671.580009:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4499, 4499] 00020000:00200000:3.0:1789760671.580018:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18427904 chunk: [18427904, 18432000) 18432000, 4096 00020000:00200000:3.0:1789760671.580024:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760671.581103:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4499, 4499] 00000008:00010000:3.0:1789760671.581139:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760671.581145:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760671.581158:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18427904, 18432000) append 0 append lockpos 0 00000080:00200000:3.0:1789760671.581198:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18427904=0x1193000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760671.581262:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760671.581265:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18432000 00000020:00010000:3.0:1789760671.581274:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760671.581277:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.581280:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.581284:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760671.581288:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760671.581298:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760671.581305:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760671.581327:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760671.581456:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 18427904<>18432000 00000080:00200000:3.0:1789760671.628669:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4499, 4499] 00000080:00200000:3.0:1789760671.628698:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760671.628704:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760671.628719:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18432000 i_blocks 36871 00000020:00200000:3.0:1789760671.628723:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760671.628733:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760671.628769:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18432000, count: 4096 00000080:00200000:3.0:1789760671.628776:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18432000, bytes: 4096 00000080:00200000:3.0:1789760671.628784:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760671.628793:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760671.628795:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18432000, 18436096) 0 0 00000080:00200000:3.0:1789760671.628800:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760671.628816:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4500, 4500] 00020000:00200000:3.0:1789760671.628825:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18432000 chunk: [18432000, 18436096) 18436096, 4096 00020000:00200000:3.0:1789760671.628830:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760671.628866:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4500, 4500] 00000008:00010000:3.0:1789760671.628885:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760671.628891:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760671.628902:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18432000, 18436096) append 0 append lockpos 0 00000080:00200000:3.0:1789760671.628915:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18432000=0x1194000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760671.629039:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760671.629042:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18436096 00000020:00010000:3.0:1789760671.629052:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760671.629055:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.629058:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.629062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760671.629065:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760671.629073:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760671.629081:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760671.629100:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760671.629263:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 18432000<>18436096 00000080:00200000:3.0:1789760671.687340:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4500, 4500] 00000080:00200000:3.0:1789760671.687362:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760671.687368:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760671.687384:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18436096 i_blocks 36871 00000020:00200000:3.0:1789760671.687388:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760671.687398:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760671.687432:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18436096, count: 4096 00000080:00200000:3.0:1789760671.687440:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18436096, bytes: 4096 00000080:00200000:3.0:1789760671.687463:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760671.687472:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760671.687474:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18436096, 18440192) 0 0 00000080:00200000:3.0:1789760671.687478:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760671.687495:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4501, 4501] 00020000:00200000:3.0:1789760671.687503:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18436096 chunk: [18436096, 18440192) 18440192, 4096 00020000:00200000:3.0:1789760671.687507:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760671.687530:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4501, 4501] 00000008:00010000:3.0:1789760671.687548:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760671.687553:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760671.687564:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18436096, 18440192) append 0 append lockpos 0 00000080:00200000:3.0:1789760671.687577:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18436096=0x1195000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760671.687641:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760671.687644:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18440192 00000020:00010000:3.0:1789760671.687653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760671.687657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.687660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.687663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760671.687666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760671.687675:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760671.687683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760671.690516:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760671.690790:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 18436096<>18440192 00000080:00200000:3.0:1789760671.755008:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4501, 4501] 00000080:00200000:3.0:1789760671.755024:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760671.755031:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760671.755049:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18440192 i_blocks 36871 00000020:00200000:3.0:1789760671.755052:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760671.755120:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760671.755210:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18440192, count: 4096 00000080:00200000:3.0:1789760671.755218:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18440192, bytes: 4096 00000080:00200000:3.0:1789760671.755229:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760671.755238:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760671.755241:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18440192, 18444288) 0 0 00000080:00200000:3.0:1789760671.755246:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760671.755265:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4502, 4502] 00020000:00200000:3.0:1789760671.755273:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18440192 chunk: [18440192, 18444288) 18444288, 4096 00020000:00200000:3.0:1789760671.755279:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760671.755303:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4502, 4502] 00000008:00010000:3.0:1789760671.755322:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760671.755529:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760671.755543:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18440192, 18444288) append 0 append lockpos 0 00000080:00200000:3.0:1789760671.755558:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18440192=0x1196000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760671.755619:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760671.755623:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18444288 00000020:00010000:3.0:1789760671.755631:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760671.755635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.755638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760671.755641:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760671.755644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760671.755652:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760671.755660:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760671.755687:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760671.755813:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 18440192<>18444288 00000100:00100000:3.0:1789760671.755961:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760671.755980:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495651584:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760671.807326:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:3.0:1789760671.807343:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760671.807354:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760671.807463:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:3.0:1789760671.807495:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495651584:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.810724:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495651712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760671.850447:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971820 gen 2 00000008:00010000:3.0:1789760671.850577:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760671.850587:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760671.850596:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760671.850659:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760671.850688:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495651712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.458998:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495653376:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.507782:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971833 gen 2 00000008:00010000:3.0:1789760672.507838:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760672.507845:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760672.507854:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760672.507923:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760672.507949:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495653376:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.508778:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760672.508795:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495653504:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.591484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495653760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.640681:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971836 gen 2 00000008:00010000:3.0:1789760672.640733:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760672.640741:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760672.640753:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760672.640969:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760672.641003:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495653760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.642507:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495653888:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760672.651373:0:468:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1789760672.651396:0:468:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760672.651429:0:468:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:3.0:1789760672.685494:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760672.685508:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760672.685518:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760672.685536:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760672.685568:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495653888:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.687370:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495654016:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.740147:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971838 gen 2 00000008:00010000:3.0:1789760672.740253:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760672.740262:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760672.740274:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760672.740298:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760672.740336:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495654016:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.742423:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495654144:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.802817:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971839 gen 2 00000008:00010000:3.0:1789760672.802867:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760672.802876:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760672.802887:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760672.802974:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760672.803004:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495654144:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.805994:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495654272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760672.853904:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971840 gen 2 00000008:00010000:3.0:1789760672.853953:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760672.853962:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760672.853973:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760672.853997:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760672.854026:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495654272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.091809:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495654784:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.183146:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971844 gen 2 00000008:00010000:3.0:1789760673.183207:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760673.183213:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760673.183223:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760673.183291:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760673.183314:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495654784:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760673.282924:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4528, 4528] 00000080:00200000:3.0:1789760673.282948:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760673.282955:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760673.282973:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18550784 i_blocks 36871 00000020:00200000:3.0:1789760673.282979:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760673.282993:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760673.283027:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18550784, count: 4096 00000080:00200000:3.0:1789760673.283035:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18550784, bytes: 4096 00000080:00200000:3.0:1789760673.283049:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760673.283058:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760673.283061:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18550784, 18554880) 0 0 00000080:00200000:3.0:1789760673.283065:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760673.283083:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4529, 4529] 00020000:00200000:3.0:1789760673.283091:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18550784 chunk: [18550784, 18554880) 18554880, 4096 00020000:00200000:3.0:1789760673.283110:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760673.283143:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4529, 4529] 00000008:00010000:3.0:1789760673.283162:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760673.283203:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760673.283216:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18550784, 18554880) append 0 append lockpos 0 00000080:00200000:3.0:1789760673.283231:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18550784=0x11b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760673.283294:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760673.283297:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18554880 00000020:00010000:3.0:1789760673.283307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760673.283310:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760673.283325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760673.283329:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760673.283332:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760673.283340:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760673.283348:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760673.283368:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760673.283511:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 18550784<>18554880 00000100:00100000:3.0:1789760673.283562:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760673.283583:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495655040:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.342088:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971846 gen 2 00000008:00010000:3.0:1789760673.342214:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:3.0:1789760673.342224:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760673.342235:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760673.342270:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:3.0:1789760673.342301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495655040:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760673.409744:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4530, 4530] 00000080:00200000:3.0:1789760673.409761:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760673.409768:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760673.409792:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18558976 i_blocks 36871 00000020:00200000:3.0:1789760673.409795:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760673.409805:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760673.409845:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18558976, count: 4096 00000080:00200000:3.0:1789760673.409852:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18558976, bytes: 4096 00000080:00200000:3.0:1789760673.409862:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760673.409870:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760673.409872:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18558976, 18563072) 0 0 00000080:00200000:3.0:1789760673.409877:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760673.409895:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4531, 4531] 00020000:00200000:3.0:1789760673.409903:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18558976 chunk: [18558976, 18563072) 18563072, 4096 00020000:00200000:3.0:1789760673.409908:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760673.409927:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4531, 4531] 00000008:00010000:3.0:1789760673.409943:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760673.409948:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760673.409959:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18558976, 18563072) append 0 append lockpos 0 00000080:00200000:3.0:1789760673.409972:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18558976=0x11b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760673.410029:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760673.410032:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18563072 00000020:00010000:3.0:1789760673.410041:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760673.410044:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760673.410047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760673.410051:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760673.410054:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760673.410061:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760673.410067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760673.410085:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760673.410464:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 18558976<>18563072 00000100:00100000:3.0:1789760673.410503:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760673.410519:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495655296:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.470490:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971848 gen 2 00000008:00010000:3.0:1789760673.470538:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:3.0:1789760673.470546:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760673.470557:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760673.470581:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:3.0:1789760673.470620:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495655296:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.617701:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495655680:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.657925:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971850 gen 2 00000100:00100000:3.0:1789760673.657956:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264000 x1876699495655680/t4294971851(4294971851) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760689 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760673.658007:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760673.658014:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760673.658023:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760673.658043:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760673.658065:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495655680:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.659136:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760673.659153:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495655808:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.702037:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971852 gen 2 00000008:00010000:3.0:1789760673.702102:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760673.702111:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760673.702296:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760673.702427:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760673.702457:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495655808:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.704518:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495655936:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760673.772940:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760673.772956:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760673.772967:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760673.772995:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760673.773027:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495655936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.821502:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495656192:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.869488:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971855 gen 2 00000008:00010000:3.0:1789760673.869556:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760673.869565:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760673.869574:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760673.869600:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760673.869629:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495656192:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.870471:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495656320:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760673.953555:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760673.953571:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760673.953585:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760673.953612:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760673.953656:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495656320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760673.954736:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495656448:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.015500:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971857 gen 2 00000008:00010000:3.0:1789760674.015568:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760674.015576:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.015586:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.015613:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760674.015639:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495656448:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.053232:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760674.053252:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495656704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.110486:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971858 gen 2 00000100:00100000:3.0:1789760674.110530:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266a00 x1876699495656704/t4294971859(4294971859) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760690 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760674.110579:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760674.110586:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.110597:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.110623:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760674.110648:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495656704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.111653:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760674.111673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495656832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.174732:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971860 gen 2 00000008:00010000:3.0:1789760674.174798:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760674.174808:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.174819:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.174854:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760674.174887:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495656832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.175819:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760674.176309:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495656960:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.223231:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971861 gen 2 00000100:00100000:3.0:1789760674.320676:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495657216:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.380949:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971863 gen 2 00000008:00010000:3.0:1789760674.380998:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760674.381006:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.381017:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.381042:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760674.381073:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495657216:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.381956:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495657344:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.452815:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971864 gen 2 00000008:00010000:3.0:1789760674.452871:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760674.452879:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.452890:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.452922:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760674.452961:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495657344:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.454601:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760674.454628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495657472:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.542282:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971865 gen 2 00000008:00010000:3.0:1789760674.542400:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760674.542410:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.542422:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.542453:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760674.542482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495657472:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.543340:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495657600:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.596249:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971866 gen 2 00000008:00010000:3.0:1789760674.596302:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760674.596312:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.596323:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.596348:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760674.596375:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495657600:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.681298:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495657856:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.750721:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971868 gen 2 00000008:00010000:3.0:1789760674.750773:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760674.750782:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.750793:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.750819:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760674.750852:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495657856:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.752003:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760674.752023:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495657984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.823243:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971869 gen 2 00000008:00010000:3.0:1789760674.823298:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760674.823308:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.823322:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.823353:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760674.823387:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495657984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.825562:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495658112:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.859939:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971870 gen 2 00000008:00010000:3.0:1789760674.859990:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760674.859999:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.860010:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.860038:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760674.860069:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495658112:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760674.860997:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:3.0:1789760674.861213:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 18653184<>18657280 00000100:00100000:3.0:1789760674.861272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495658240:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760674.904483:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971871 gen 2 00000008:00010000:3.0:1789760674.904529:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760674.904539:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760674.904550:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760674.904574:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760674.904605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495658240:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760674.964570:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4555, 4555] 00000080:00200000:3.0:1789760674.964585:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760674.964591:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760674.964606:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18661376 i_blocks 36871 00000020:00200000:3.0:1789760674.964610:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760674.964620:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760674.965288:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18661376, count: 4096 00000080:00200000:3.0:1789760674.965307:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18661376, bytes: 4096 00000080:00200000:3.0:1789760674.965428:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760674.965437:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760674.965439:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18661376, 18665472) 0 0 00000080:00200000:3.0:1789760674.965444:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760674.965539:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4556, 4556] 00020000:00200000:3.0:1789760674.965547:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18661376 chunk: [18661376, 18665472) 18665472, 4096 00020000:00200000:3.0:1789760674.965551:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760674.965568:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4556, 4556] 00000008:00010000:3.0:1789760674.965583:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760674.965588:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760674.965596:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18661376, 18665472) append 0 append lockpos 0 00000080:00200000:3.0:1789760674.965606:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18661376=0x11cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760674.965656:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760674.965658:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18665472 00000020:00010000:3.0:1789760674.965664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760674.965667:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760674.965669:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760674.965672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760674.965674:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760674.965682:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760674.965687:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760674.965709:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760674.965804:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 18661376<>18665472 00000080:00200000:3.0:1789760675.023135:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4556, 4556] 00000080:00200000:3.0:1789760675.023158:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760675.023164:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760675.023206:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18665472 i_blocks 36871 00000020:00200000:3.0:1789760675.023210:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760675.023222:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760675.023247:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18665472, count: 4096 00000080:00200000:3.0:1789760675.023253:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18665472, bytes: 4096 00000080:00200000:3.0:1789760675.023264:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760675.023272:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760675.023274:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18665472, 18669568) 0 0 00000080:00200000:3.0:1789760675.023278:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760675.023292:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4557, 4557] 00020000:00200000:3.0:1789760675.023299:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18665472 chunk: [18665472, 18669568) 18669568, 4096 00020000:00200000:3.0:1789760675.023303:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760675.023332:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4557, 4557] 00000008:00010000:3.0:1789760675.023348:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760675.023354:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760675.023362:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18665472, 18669568) append 0 append lockpos 0 00000080:00200000:3.0:1789760675.023376:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18665472=0x11cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760675.023438:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760675.023441:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18669568 00000020:00010000:3.0:1789760675.023456:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760675.023460:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.023462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.023465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760675.023468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760675.023475:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760675.023480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760675.023498:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760675.023598:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 18665472<>18669568 00000080:00200000:3.0:1789760675.080661:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4557, 4557] 00000080:00200000:3.0:1789760675.080677:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760675.080684:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760675.080699:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18669568 i_blocks 36871 00000020:00200000:3.0:1789760675.080703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760675.080713:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760675.080748:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18669568, count: 4096 00000080:00200000:3.0:1789760675.080755:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18669568, bytes: 4096 00000080:00200000:3.0:1789760675.080765:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760675.080774:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760675.080776:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18669568, 18673664) 0 0 00000080:00200000:3.0:1789760675.080781:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760675.080798:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4558, 4558] 00020000:00200000:3.0:1789760675.080806:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18669568 chunk: [18669568, 18673664) 18673664, 4096 00020000:00200000:3.0:1789760675.080811:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760675.080832:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4558, 4558] 00000008:00010000:3.0:1789760675.080859:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760675.080865:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760675.080877:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18669568, 18673664) append 0 append lockpos 0 00000080:00200000:3.0:1789760675.080891:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18669568=0x11ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760675.080952:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760675.080954:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18673664 00000020:00010000:3.0:1789760675.080963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760675.080966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.080971:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.080974:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760675.080977:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760675.080985:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760675.080993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760675.081010:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760675.081150:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 18669568<>18673664 00000080:00200000:3.0:1789760675.163535:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4558, 4558] 00000080:00200000:3.0:1789760675.163554:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760675.163561:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760675.163578:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18673664 i_blocks 36871 00000020:00200000:3.0:1789760675.163582:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760675.163592:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760675.163628:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18673664, count: 4096 00000080:00200000:3.0:1789760675.163636:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18673664, bytes: 4096 00000080:00200000:3.0:1789760675.163647:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760675.163656:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760675.163658:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18673664, 18677760) 0 0 00000080:00200000:3.0:1789760675.163663:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760675.163682:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4559, 4559] 00020000:00200000:3.0:1789760675.163691:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18673664 chunk: [18673664, 18677760) 18677760, 4096 00020000:00200000:3.0:1789760675.163697:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760675.163719:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4559, 4559] 00000008:00010000:3.0:1789760675.163738:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760675.163744:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760675.163755:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18673664, 18677760) append 0 append lockpos 0 00000080:00200000:3.0:1789760675.163770:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18673664=0x11cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760675.163828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760675.163832:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18677760 00000020:00010000:3.0:1789760675.163840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760675.163844:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.163847:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.163850:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760675.163853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760675.163861:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760675.163868:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760675.163886:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760675.164013:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 18673664<>18677760 00000080:00200000:3.0:1789760675.210541:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4559, 4559] 00000080:00200000:3.0:1789760675.210559:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760675.210566:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760675.210581:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18677760 i_blocks 36871 00000020:00200000:3.0:1789760675.210585:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760675.210595:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760675.210630:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18677760, count: 4096 00000080:00200000:3.0:1789760675.210638:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18677760, bytes: 4096 00000080:00200000:3.0:1789760675.210648:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760675.210656:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760675.210659:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18677760, 18681856) 0 0 00000080:00200000:3.0:1789760675.210664:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760675.210682:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4560, 4560] 00020000:00200000:3.0:1789760675.210690:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18677760 chunk: [18677760, 18681856) 18681856, 4096 00020000:00200000:3.0:1789760675.210694:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760675.210715:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4560, 4560] 00000008:00010000:3.0:1789760675.210733:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760675.210738:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760675.210749:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18677760, 18681856) append 0 append lockpos 0 00000080:00200000:3.0:1789760675.210763:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18677760=0x11d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760675.210812:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760675.210815:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18681856 00000020:00010000:3.0:1789760675.210822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760675.210825:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.210828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.210831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760675.210834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760675.210841:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760675.210847:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760675.291573:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4560, 4560] 00000080:00200000:3.0:1789760675.291589:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760675.291596:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760675.291610:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18681856 i_blocks 36871 00000020:00200000:3.0:1789760675.291614:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760675.291622:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760675.291655:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18681856, count: 4096 00000080:00200000:3.0:1789760675.291662:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18681856, bytes: 4096 00000080:00200000:3.0:1789760675.291672:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760675.291681:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760675.291684:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18681856, 18685952) 0 0 00000080:00200000:3.0:1789760675.291689:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760675.291704:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4561, 4561] 00020000:00200000:3.0:1789760675.291713:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18681856 chunk: [18681856, 18685952) 18685952, 4096 00020000:00200000:3.0:1789760675.291717:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760675.291736:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4561, 4561] 00000008:00010000:3.0:1789760675.291752:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760675.291757:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760675.291769:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18681856, 18685952) append 0 append lockpos 0 00000080:00200000:3.0:1789760675.291782:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18681856=0x11d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760675.291842:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760675.291845:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18685952 00000020:00010000:3.0:1789760675.291866:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760675.291870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.291873:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.291876:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760675.291879:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760675.291888:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760675.291895:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760675.291912:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760675.292041:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 18681856<>18685952 00000080:00200000:3.0:1789760675.373896:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4561, 4561] 00000080:00200000:3.0:1789760675.373915:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760675.373921:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760675.373937:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18685952 i_blocks 36871 00000020:00200000:3.0:1789760675.373940:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760675.373950:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760675.373984:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18685952, count: 4096 00000080:00200000:3.0:1789760675.373992:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18685952, bytes: 4096 00000080:00200000:3.0:1789760675.374001:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760675.374010:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760675.374013:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18685952, 18690048) 0 0 00000080:00200000:3.0:1789760675.374018:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760675.374035:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4562, 4562] 00020000:00200000:3.0:1789760675.374044:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18685952 chunk: [18685952, 18690048) 18690048, 4096 00020000:00200000:3.0:1789760675.374049:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760675.374069:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4562, 4562] 00000008:00010000:3.0:1789760675.374088:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760675.374094:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760675.374115:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18685952, 18690048) append 0 append lockpos 0 00000080:00200000:3.0:1789760675.374131:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18685952=0x11d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760675.374746:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760675.374751:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18690048 00000020:00010000:3.0:1789760675.374815:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760675.374820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.374823:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760675.374827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760675.374830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760675.374838:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760675.374845:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760675.374869:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760675.375623:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 18685952<>18690048 00000100:00100000:3.0:1789760675.623220:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495659776:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760675.672628:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971883 gen 2 00000008:00010000:3.0:1789760675.672700:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760675.672708:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760675.672719:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760675.672747:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760675.672778:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495659776:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760675.673817:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495659904:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760675.755742:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760675.755758:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760675.755771:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760675.755791:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760675.755826:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495659904:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760675.911727:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495660416:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760675.961610:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760675.961627:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760675.961639:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760675.961669:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760675.961701:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495660416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760675.966337:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760675.966366:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495660544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.041130:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971889 gen 2 00000008:00010000:3.0:1789760676.041242:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760676.041255:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760676.041267:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760676.041290:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760676.041321:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495660544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.042085:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760676.042116:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495660672:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.100034:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971890 gen 2 00000008:00010000:3.0:1789760676.100088:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760676.100104:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760676.100114:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760676.100132:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760676.100162:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495660672:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.101480:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760676.101501:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495660800:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.148292:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971891 gen 2 00000008:00010000:3.0:1789760676.148343:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760676.148352:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760676.148363:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760676.148528:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760676.148556:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495660800:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.149602:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495660928:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.544532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495662208:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760676.586537:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760676.586552:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760676.586565:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760676.586590:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760676.586621:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495662208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.587630:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760676.587649:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495662336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.642970:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971903 gen 2 00000008:00010000:3.0:1789760676.643035:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760676.643044:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760676.643064:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760676.643100:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760676.643130:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495662336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.644737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495662464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760676.683245:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971904 gen 2 00000008:00010000:3.0:1789760676.683293:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760676.683300:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760676.683310:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760676.683331:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760676.683357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495662464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760677.095209:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495664256:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760677.254610:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4601, 4601] 00000080:00200000:3.0:1789760677.254627:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760677.254634:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760677.254650:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18849792 i_blocks 36871 00000020:00200000:3.0:1789760677.254654:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760677.254664:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760677.254698:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18849792, count: 4096 00000080:00200000:3.0:1789760677.254705:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18849792, bytes: 4096 00000080:00200000:3.0:1789760677.254715:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760677.254724:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760677.254726:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18849792, 18853888) 0 0 00000080:00200000:3.0:1789760677.254731:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760677.254749:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4602, 4602] 00020000:00200000:3.0:1789760677.254757:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18849792 chunk: [18849792, 18853888) 18853888, 4096 00020000:00200000:3.0:1789760677.254762:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760677.254783:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4602, 4602] 00000008:00010000:3.0:1789760677.254802:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760677.254808:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760677.254819:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18849792, 18853888) append 0 append lockpos 0 00000080:00200000:3.0:1789760677.254833:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18849792=0x11fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760677.254904:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760677.254907:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18853888 00000020:00010000:3.0:1789760677.254917:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760677.254920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760677.254924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760677.254927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760677.254930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760677.255287:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760677.255303:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760677.255333:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760677.255443:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 18849792<>18853888 00000100:00100000:3.0:1789760677.255493:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760677.255514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495664768:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760677.305842:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971919 gen 2 00000008:00010000:3.0:1789760677.305903:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760677.305913:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760677.305926:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760677.305957:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760677.305987:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495664768:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760677.717545:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4611, 4611] 00000080:00200000:3.0:1789760677.717561:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760677.717567:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760677.717583:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18890752 i_blocks 38919 00000020:00200000:3.0:1789760677.717587:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760677.717597:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760677.717645:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18890752, count: 4096 00000080:00200000:3.0:1789760677.717653:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18890752, bytes: 4096 00000080:00200000:3.0:1789760677.717664:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760677.717673:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760677.717676:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18890752, 18894848) 0 0 00000080:00200000:3.0:1789760677.717681:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760677.717697:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4612, 4612] 00020000:00200000:3.0:1789760677.717705:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18890752 chunk: [18890752, 18894848) 18894848, 4096 00020000:00200000:3.0:1789760677.717711:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760677.717732:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4612, 4612] 00000008:00010000:3.0:1789760677.717751:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760677.717757:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760677.717769:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18890752, 18894848) append 0 append lockpos 0 00000080:00200000:3.0:1789760677.717785:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18890752=0x1204000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760677.717844:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760677.717847:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18894848 00000020:00010000:3.0:1789760677.717870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760677.717874:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760677.717877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760677.717881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760677.717883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760677.717891:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760677.717899:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760677.717916:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760677.718039:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 18890752<>18894848 00000100:00100000:3.0:1789760677.718097:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760677.718114:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495666048:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760677.766992:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760677.767008:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760677.767020:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760677.767055:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760677.767099:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495666048:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760677.846452:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4613, 4613] 00000080:00200000:3.0:1789760677.846478:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760677.846483:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760677.846501:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18898944 i_blocks 38919 00000020:00200000:3.0:1789760677.846504:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760677.846514:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760677.846564:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18898944, count: 4096 00000080:00200000:3.0:1789760677.846571:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18898944, bytes: 4096 00000080:00200000:3.0:1789760677.846581:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760677.846591:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760677.846593:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18898944, 18903040) 0 0 00000080:00200000:3.0:1789760677.846598:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760677.846615:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4614, 4614] 00020000:00200000:3.0:1789760677.846623:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 18898944 chunk: [18898944, 18903040) 18903040, 4096 00020000:00200000:3.0:1789760677.846627:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760677.846646:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4614, 4614] 00000008:00010000:3.0:1789760677.846663:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760677.846669:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760677.846679:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [18898944, 18903040) append 0 append lockpos 0 00000080:00200000:3.0:1789760677.846691:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=18898944=0x1206000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760677.846748:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760677.846751:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 18903040 00000020:00010000:3.0:1789760677.846759:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760677.846763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760677.846766:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760677.846769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd206587038 (ffff8dd218250ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760677.846772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd218250ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760677.846780:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760677.846787:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760677.846803:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760677.846921:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 18898944<>18903040 00000100:00100000:3.0:1789760677.846965:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760677.846980:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495666304:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760677.883671:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760677.883687:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760677.883699:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760677.883731:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760677.883763:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495666304:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760677.934928:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4615, 4615] 00000080:00200000:3.0:1789760677.934945:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760677.934952:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760677.934969:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 18907136 i_blocks 38919 00000020:00200000:3.0:1789760677.934973:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760677.934992:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760677.935037:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 18907136, count: 4096 00000080:00200000:3.0:1789760677.935047:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 18907136, bytes: 4096 00000080:00200000:3.0:1789760677.936124:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760677.936139:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760677.936142:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [18907136, 18911232) 0 0 00000080:00200000:3.0:1789760677.936147:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000100:00100000:3.0:1789760677.938221:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495666560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760679.102064:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495669632:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760679.147211:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971957 gen 2 00000008:00010000:3.0:1789760679.147277:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760679.147286:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760679.147297:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760679.147320:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760679.147350:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495669632:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760679.149364:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495669760:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760679.192807:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760679.192824:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760679.192836:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760679.192868:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760679.192898:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495669760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760679.194699:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495669888:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760679.276416:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971958 gen 2 00000100:00100000:3.0:1789760679.276459:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264700 x1876699495669888/t4294971959(4294971959) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760695 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789760679.283298:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4642, 4642] 00000080:00200000:3.0:1789760679.283329:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760679.283338:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760679.283359:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19017728 i_blocks 38919 00000020:00200000:3.0:1789760679.283366:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760679.283385:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760679.283464:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19017728, count: 4096 00000080:00200000:3.0:1789760679.283474:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19017728, bytes: 4096 00000080:00200000:3.0:1789760679.283490:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760679.283500:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760679.283502:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19017728, 19021824) 0 0 00000080:00200000:3.0:1789760679.283507:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760679.283526:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4643, 4643] 00020000:00200000:3.0:1789760679.283535:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19017728 chunk: [19017728, 19021824) 19021824, 4096 00020000:00200000:3.0:1789760679.283541:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760679.283565:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4643, 4643] 00000008:00010000:3.0:1789760679.283598:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760679.283604:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760679.283618:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19017728, 19021824) append 0 append lockpos 0 00000080:00200000:3.0:1789760679.283644:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19017728=0x1223000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760679.283715:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760679.283717:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19021824 00000020:00010000:3.0:1789760679.283726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760679.283729:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760679.283733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760679.283737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760679.283740:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760679.283748:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760679.283755:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760679.283778:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760679.283916:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 19017728<>19021824 00000080:00200000:3.0:1789760679.350491:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4643, 4643] 00000080:00200000:3.0:1789760679.350509:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760679.350515:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760679.350530:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19021824 i_blocks 38919 00000020:00200000:3.0:1789760679.350534:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760679.350544:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760679.350593:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19021824, count: 4096 00000080:00200000:3.0:1789760679.350600:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19021824, bytes: 4096 00000080:00200000:3.0:1789760679.350616:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760679.350627:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760679.350629:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19021824, 19025920) 0 0 00000080:00200000:3.0:1789760679.350634:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760679.350650:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4644, 4644] 00020000:00200000:3.0:1789760679.350658:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19021824 chunk: [19021824, 19025920) 19025920, 4096 00020000:00200000:3.0:1789760679.350663:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760679.350681:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4644, 4644] 00000008:00010000:3.0:1789760679.350697:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760679.350702:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760679.350712:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19021824, 19025920) append 0 append lockpos 0 00000080:00200000:3.0:1789760679.350727:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19021824=0x1224000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760679.350784:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760679.350787:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19025920 00000020:00010000:3.0:1789760679.350795:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760679.350798:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760679.350803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760679.350806:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760679.350808:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760679.350815:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760679.350822:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760679.350839:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760679.351694:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 19021824<>19025920 00000100:00100000:3.0:1789760679.351753:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760679.351775:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495670144:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760679.639041:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4646, 4646] 00000080:00200000:3.0:1789760679.639100:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760679.639107:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760679.639124:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19034112 i_blocks 38919 00000020:00200000:3.0:1789760679.639130:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760679.639144:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760679.639463:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19034112, count: 4096 00000080:00200000:3.0:1789760679.639473:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19034112, bytes: 4096 00000080:00200000:3.0:1789760679.639484:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760679.639492:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760679.639495:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19034112, 19038208) 0 0 00000080:00200000:3.0:1789760679.639500:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760679.639517:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4647, 4647] 00020000:00200000:3.0:1789760679.639525:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19034112 chunk: [19034112, 19038208) 19038208, 4096 00020000:00200000:3.0:1789760679.639529:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760679.639551:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4647, 4647] 00000008:00010000:3.0:1789760679.639569:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760679.639575:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760679.639587:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19034112, 19038208) append 0 append lockpos 0 00000080:00200000:3.0:1789760679.639600:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19034112=0x1227000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760679.640338:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760679.640344:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19038208 00000020:00010000:3.0:1789760679.640354:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760679.640358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760679.640362:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760679.640365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760679.640368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760679.640379:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760679.640386:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760679.640520:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760679.641313:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 19034112<>19038208 00000100:00100000:3.0:1789760679.641381:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760679.641984:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495670528:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760679.733683:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971964 gen 2 00000008:00010000:3.0:1789760679.733733:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:3.0:1789760679.733741:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760679.733753:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760679.733829:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:3.0:1789760679.733870:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495670528:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760680.046966:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4650, 4650] 00000080:00200000:3.0:1789760680.046982:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760680.046988:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760680.047003:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19050496 i_blocks 38919 00000020:00200000:3.0:1789760680.047007:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760680.047015:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760680.047063:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19050496, count: 4096 00000080:00200000:3.0:1789760680.047079:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19050496, bytes: 4096 00000080:00200000:3.0:1789760680.047099:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760680.047108:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760680.047110:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19050496, 19054592) 0 0 00000080:00200000:3.0:1789760680.047116:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760680.047131:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4651, 4651] 00020000:00200000:3.0:1789760680.047139:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19050496 chunk: [19050496, 19054592) 19054592, 4096 00020000:00200000:3.0:1789760680.047144:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760680.047162:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4651, 4651] 00000008:00010000:3.0:1789760680.047211:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760680.047217:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760680.047228:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19050496, 19054592) append 0 append lockpos 0 00000080:00200000:3.0:1789760680.047243:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19050496=0x122b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760680.047307:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760680.047310:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19054592 00000020:00010000:3.0:1789760680.047318:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760680.047322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760680.047325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760680.047329:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760680.047332:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760680.047339:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760680.047345:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760680.047360:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760680.047486:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 19050496<>19054592 00000080:00200000:3.0:1789760680.087857:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4651, 4651] 00000080:00200000:3.0:1789760680.087873:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760680.087879:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760680.087894:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19054592 i_blocks 38919 00000020:00200000:3.0:1789760680.087897:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760680.087906:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760680.087938:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19054592, count: 4096 00000080:00200000:3.0:1789760680.087945:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19054592, bytes: 4096 00000080:00200000:3.0:1789760680.087955:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760680.087964:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760680.087966:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19054592, 19058688) 0 0 00000080:00200000:3.0:1789760680.087971:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760680.087987:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4652, 4652] 00020000:00200000:3.0:1789760680.087995:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19054592 chunk: [19054592, 19058688) 19058688, 4096 00020000:00200000:3.0:1789760680.088000:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760680.088019:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4652, 4652] 00000008:00010000:3.0:1789760680.088036:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760680.088043:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760680.088054:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19054592, 19058688) append 0 append lockpos 0 00000080:00200000:3.0:1789760680.088076:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19054592=0x122c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760680.088132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760680.088135:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19058688 00000020:00010000:3.0:1789760680.088143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760680.088146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760680.088150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760680.088153:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760680.088156:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760680.088163:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760680.088190:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760680.088207:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760680.088582:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 19054592<>19058688 00000080:00200000:3.0:1789760680.135008:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4652, 4652] 00000080:00200000:3.0:1789760680.135024:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760680.135031:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760680.135046:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19058688 i_blocks 38919 00000020:00200000:3.0:1789760680.135050:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760680.135078:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760680.135110:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19058688, count: 4096 00000080:00200000:3.0:1789760680.135118:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19058688, bytes: 4096 00000080:00200000:3.0:1789760680.135127:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760680.135136:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760680.135138:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19058688, 19062784) 0 0 00000080:00200000:3.0:1789760680.135143:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760680.135160:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4653, 4653] 00020000:00200000:3.0:1789760680.135201:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19058688 chunk: [19058688, 19062784) 19062784, 4096 00020000:00200000:3.0:1789760680.135206:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760680.135226:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4653, 4653] 00000008:00010000:3.0:1789760680.135244:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760680.135250:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760680.135263:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19058688, 19062784) append 0 append lockpos 0 00000080:00200000:3.0:1789760680.135278:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19058688=0x122d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760680.135343:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760680.135345:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19062784 00000020:00010000:3.0:1789760680.135354:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760680.135358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760680.135361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760680.135365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760680.135368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760680.135375:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760680.135383:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760680.135627:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760680.135759:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 19058688<>19062784 00000080:00200000:3.0:1789760680.179825:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4653, 4653] 00000080:00200000:3.0:1789760680.179840:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760680.179847:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760680.179871:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19062784 i_blocks 38919 00000020:00200000:3.0:1789760680.179875:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760680.179883:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760680.179915:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19062784, count: 4096 00000080:00200000:3.0:1789760680.179921:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19062784, bytes: 4096 00000080:00200000:3.0:1789760680.179931:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760680.179939:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760680.179942:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19062784, 19066880) 0 0 00000080:00200000:3.0:1789760680.179947:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760680.179963:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4654, 4654] 00020000:00200000:3.0:1789760680.179971:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19062784 chunk: [19062784, 19066880) 19066880, 4096 00020000:00200000:3.0:1789760680.179975:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760680.179992:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4654, 4654] 00000008:00010000:3.0:1789760680.180008:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760680.180013:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760680.180024:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19062784, 19066880) append 0 append lockpos 0 00000080:00200000:3.0:1789760680.180038:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19062784=0x122e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760680.180109:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760680.180112:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19066880 00000020:00010000:3.0:1789760680.180120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760680.180123:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760680.180126:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760680.180130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760680.180133:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760680.180140:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760680.180146:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760680.180162:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760680.180687:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 19062784<>19066880 00000100:00100000:3.0:1789760680.180727:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760680.180743:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495671424:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760680.220830:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:3.0:1789760680.220922:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760680.220936:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760680.221015:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:3.0:1789760680.221055:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495671424:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.663660:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495672320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.697117:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971977 gen 2 00000100:00100000:3.0:1789760680.697159:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264700 x1876699495672320/t4294971978(4294971978) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760696 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760680.697440:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760680.697448:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760680.697460:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760680.697480:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760680.697510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495672320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.698097:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495672448:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.735280:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971979 gen 2 00000008:00010000:3.0:1789760680.735344:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760680.735353:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760680.735364:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760680.735390:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760680.735420:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495672448:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.738282:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760680.738310:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495672576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.780798:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971980 gen 2 00000008:00010000:3.0:1789760680.780913:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760680.780923:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760680.780935:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760680.780962:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760680.780994:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495672576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.782076:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495672704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.830854:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971981 gen 2 00000008:00010000:3.0:1789760680.830910:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760680.830921:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760680.830936:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760680.830966:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760680.831004:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495672704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.831862:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495672832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.895684:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971982 gen 2 00000008:00010000:3.0:1789760680.895736:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760680.895745:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760680.895757:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760680.895786:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760680.895818:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495672832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760680.942378:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760680.946418:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495673088:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760681.028452:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971984 gen 2 00000008:00010000:3.0:1789760681.028503:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760681.028510:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760681.028522:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760681.028548:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760681.028577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495673088:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760681.130109:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4668, 4668] 00000080:00200000:3.0:1789760681.130125:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760681.130131:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760681.130155:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19124224 i_blocks 38919 00000020:00200000:3.0:1789760681.130158:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760681.130198:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760681.130234:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19124224, count: 4096 00000080:00200000:3.0:1789760681.130240:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19124224, bytes: 4096 00000080:00200000:3.0:1789760681.130252:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760681.130259:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760681.130262:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19124224, 19128320) 0 0 00000080:00200000:3.0:1789760681.130266:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760681.130282:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4669, 4669] 00020000:00200000:3.0:1789760681.130290:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19124224 chunk: [19124224, 19128320) 19128320, 4096 00020000:00200000:3.0:1789760681.130294:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760681.130314:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4669, 4669] 00000008:00010000:3.0:1789760681.130334:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760681.130340:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760681.130352:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19124224, 19128320) append 0 append lockpos 0 00000080:00200000:3.0:1789760681.130366:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19124224=0x123d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760681.130698:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760681.130702:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19128320 00000020:00010000:3.0:1789760681.130712:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760681.130716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760681.130719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760681.130722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760681.130724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760681.130733:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760681.130740:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760681.130761:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760681.130955:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 19124224<>19128320 00000080:00200000:3.0:1789760681.282201:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4669, 4669] 00000080:00200000:3.0:1789760681.282222:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760681.282228:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760681.282245:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19128320 i_blocks 38919 00000020:00200000:3.0:1789760681.282249:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760681.282259:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760681.282293:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19128320, count: 4096 00000080:00200000:3.0:1789760681.282301:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19128320, bytes: 4096 00000080:00200000:3.0:1789760681.282314:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760681.282325:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760681.282328:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19128320, 19132416) 0 0 00000080:00200000:3.0:1789760681.282333:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760681.282353:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4670, 4670] 00020000:00200000:3.0:1789760681.282364:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19128320 chunk: [19128320, 19132416) 19132416, 4096 00020000:00200000:3.0:1789760681.282370:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760681.282493:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4670, 4670] 00000008:00010000:3.0:1789760681.282511:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760681.282517:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760681.282529:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19128320, 19132416) append 0 append lockpos 0 00000080:00200000:3.0:1789760681.282545:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19128320=0x123e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760681.282609:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760681.282612:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19132416 00000020:00010000:3.0:1789760681.282621:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760681.282624:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760681.282627:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760681.282630:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760681.282633:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760681.282641:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760681.282647:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760681.282667:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760681.282793:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 19128320<>19132416 00000100:00100000:3.0:1789760681.706942:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760681.706963:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495673984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760681.788224:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971991 gen 2 00000008:00010000:3.0:1789760681.788278:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760681.788288:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760681.788299:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760681.788324:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760681.788351:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495673984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760681.790692:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495674112:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760681.858295:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971992 gen 2 00000008:00010000:3.0:1789760681.858361:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760681.858369:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760681.858392:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760681.858416:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760681.858446:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495674112:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760681.868267:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495674496:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760681.868417:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760681.868425:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2359:1876699495674368:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760681.868464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2359:1876699495674624:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760681.872731:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2359:1876699495674368:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760681.872760:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2359:1876699495674624:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760681.872774:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495674496:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760681.931151:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495674752:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760681.982221:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760681.982237:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760681.982249:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760681.982270:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760681.982299:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495674752:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760681.984583:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760681.984632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495674880:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760682.038941:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971995 gen 2 00000008:00010000:3.0:1789760682.038999:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760682.039007:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760682.039017:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760682.039040:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760682.039077:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495674880:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760682.045402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495675008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760682.096314:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971996 gen 2 00000008:00010000:3.0:1789760682.097266:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760682.097276:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760682.097288:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760682.097313:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760682.097348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495675008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760682.100072:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495675136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760682.166297:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294971997 gen 2 00000008:00010000:3.0:1789760682.166364:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813780 00000020:00200000:3.0:1789760682.166373:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760682.166384:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760682.166409:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813780 00000100:00100000:3.0:1789760682.166448:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495675136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.191550:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495677696:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.239361:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972017 gen 2 00000008:00010000:3.0:1789760683.239498:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760683.239508:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760683.239520:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760683.239544:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760683.239575:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495677696:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.307751:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495677952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.393352:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972018 gen 2 00000100:00100000:3.0:1789760683.393535:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265880 x1876699495677952/t4294972019(4294972019) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760699 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760683.393600:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760683.393609:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760683.393621:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760683.393651:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760683.393679:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495677952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.396757:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760683.396791:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495678080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.510131:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972020 gen 2 00000008:00010000:3.0:1789760683.510229:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760683.510238:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760683.510249:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760683.510274:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760683.510305:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495678080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.510938:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760683.510954:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495678208:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760683.568878:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760683.568893:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760683.568904:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760683.568932:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760683.568961:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495678208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.570303:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760683.570325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495678336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.622414:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972022 gen 2 00000008:00010000:3.0:1789760683.622483:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760683.622492:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760683.622501:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760683.622528:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760683.622557:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495678336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.623480:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760683.623499:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495678464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.676991:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972023 gen 2 00000008:00010000:3.0:1789760683.677057:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760683.677066:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760683.677078:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760683.677104:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760683.677139:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495678464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.678001:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760683.678021:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495678592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.733512:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972024 gen 2 00000008:00010000:3.0:1789760683.733562:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760683.733571:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760683.733584:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760683.733607:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760683.733637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495678592:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760683.739974:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19283968, 19288064) append 0 append lockpos 0 00000080:00200000:3.0:1789760683.740009:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19283968=0x1264000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760683.740275:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760683.740280:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19288064 00000020:00010000:3.0:1789760683.740291:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760683.740294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760683.740299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d3b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760683.740302:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760683.740305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d3b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760683.740318:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760683.740328:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760683.740353:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760683.740892:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 19283968<>19288064 00000080:00200000:3.0:1789760683.801593:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4708, 4708] 00000080:00200000:3.0:1789760683.801637:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760683.801839:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760683.897408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495679104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760683.950484:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972028 gen 2 00000008:00010000:3.0:1789760683.950532:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760683.950541:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760683.950553:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760683.950569:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760683.950590:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495679104:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760683.979328:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4712, 4712] 00000080:00200000:3.0:1789760683.979344:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760683.979350:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760683.979366:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19304448 i_blocks 38919 00000020:00200000:3.0:1789760683.979370:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760683.979381:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760683.979443:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19304448, count: 4096 00000080:00200000:3.0:1789760683.979451:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19304448, bytes: 4096 00000080:00200000:3.0:1789760683.979460:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760683.979469:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760683.979472:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19304448, 19308544) 0 0 00000080:00200000:3.0:1789760683.979476:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760683.979492:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4713, 4713] 00020000:00200000:3.0:1789760683.979500:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19304448 chunk: [19304448, 19308544) 19308544, 4096 00020000:00200000:3.0:1789760683.979505:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760683.979525:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4713, 4713] 00000008:00010000:3.0:1789760683.979543:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760683.979548:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760683.979559:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19304448, 19308544) append 0 append lockpos 0 00000080:00200000:3.0:1789760683.979572:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19304448=0x1269000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760683.979628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760683.979631:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19308544 00000020:00010000:3.0:1789760683.979639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760683.979642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760683.979645:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760683.979649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760683.979651:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760683.979659:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760683.979665:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760683.979680:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760683.979869:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 19304448<>19308544 00000100:00100000:3.0:1789760683.980434:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760683.980456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495679360:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760684.058664:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:3.0:1789760684.058680:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760684.058693:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760684.058844:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:3.0:1789760684.058874:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495679360:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760685.368979:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4730, 4730] 00000080:00200000:3.0:1789760685.369008:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760685.369015:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760685.369045:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19378176 i_blocks 38919 00000020:00200000:3.0:1789760685.369051:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760685.369070:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760685.369109:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19378176, count: 4096 00000080:00200000:3.0:1789760685.369117:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19378176, bytes: 4096 00000080:00200000:3.0:1789760685.369133:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760685.369143:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760685.369145:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19378176, 19382272) 0 0 00000080:00200000:3.0:1789760685.369150:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760685.369195:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4731, 4731] 00020000:00200000:3.0:1789760685.369205:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19378176 chunk: [19378176, 19382272) 19382272, 4096 00020000:00200000:3.0:1789760685.369209:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760685.369235:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4731, 4731] 00000008:00010000:3.0:1789760685.369257:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760685.369262:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760685.369276:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19378176, 19382272) append 0 append lockpos 0 00000080:00200000:3.0:1789760685.369293:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19378176=0x127b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760685.369371:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760685.369375:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19382272 00000020:00010000:3.0:1789760685.369385:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760685.369389:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760685.369392:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760685.369397:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760685.369399:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760685.369408:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760685.369416:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760685.369440:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760685.369595:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 19378176<>19382272 00000100:00100000:3.0:1789760685.369659:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760685.369830:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495681664:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.525146:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972048 gen 2 00000008:00010000:3.0:1789760685.525227:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:3.0:1789760685.525235:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760685.525246:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760685.525269:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:3.0:1789760685.525295:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495681664:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.665948:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682048:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.809965:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972050 gen 2 00000100:00100000:3.0:1789760685.810015:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205267480 x1876699495682048/t4294972051(4294972051) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760701 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760685.810075:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760685.810082:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760685.810093:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760685.810116:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760685.810138:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682048:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.811104:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760685.811123:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682176:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.879833:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972052 gen 2 00000008:00010000:3.0:1789760685.879908:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760685.879918:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760685.879928:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760685.879952:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760685.879982:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682176:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.880968:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682304:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.948253:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972053 gen 2 00000008:00010000:3.0:1789760685.948306:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760685.948315:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760685.948327:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760685.948413:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760685.948446:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682304:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.949693:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760685.949714:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682432:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.982311:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972054 gen 2 00000008:00010000:3.0:1789760685.982372:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760685.982380:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760685.982390:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760685.982413:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760685.982442:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682432:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760685.983318:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.020034:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972055 gen 2 00000008:00010000:3.0:1789760686.020083:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760686.020092:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760686.020103:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760686.020206:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760686.020237:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.021502:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760686.021523:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682688:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.059465:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972056 gen 2 00000008:00010000:3.0:1789760686.059529:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760686.059537:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760686.059548:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760686.059570:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760686.059609:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682688:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.061126:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682816:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760686.112775:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760686.112790:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760686.112804:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760686.112825:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760686.112860:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.113685:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760686.113702:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682944:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.164141:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972058 gen 2 00000008:00010000:3.0:1789760686.164776:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760686.164857:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760686.164869:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760686.164901:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760686.164934:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495682944:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.167104:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495683072:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760686.217860:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4742, 4742] 00000080:00200000:3.0:1789760686.217879:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760686.217886:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760686.217901:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19427328 i_blocks 38919 00000020:00200000:3.0:1789760686.217905:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760686.217915:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760686.217950:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19427328, count: 4096 00000080:00200000:3.0:1789760686.217957:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19427328, bytes: 4096 00000080:00200000:3.0:1789760686.217968:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760686.217976:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760686.217979:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19427328, 19431424) 0 0 00000080:00200000:3.0:1789760686.217984:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760686.218001:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4743, 4743] 00020000:00200000:3.0:1789760686.218009:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19427328 chunk: [19427328, 19431424) 19431424, 4096 00020000:00200000:3.0:1789760686.218013:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760686.218046:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4743, 4743] 00000008:00010000:3.0:1789760686.218066:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760686.218072:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760686.218084:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19427328, 19431424) append 0 append lockpos 0 00000080:00200000:3.0:1789760686.218098:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19427328=0x1287000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760686.218158:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760686.218161:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19431424 00000020:00010000:3.0:1789760686.218188:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760686.218192:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760686.218196:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760686.218199:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760686.218202:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760686.218212:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760686.218220:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760686.218237:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760686.218367:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 19427328<>19431424 00000100:00100000:3.0:1789760686.296288:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972060 gen 2 00000008:00010000:3.0:1789760686.296355:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:3.0:1789760686.296364:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760686.296378:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760686.296402:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:3.0:1789760686.296434:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495683200:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760686.449714:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4745, 4745] 00000080:00200000:3.0:1789760686.449731:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760686.449737:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760686.449753:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19439616 i_blocks 38919 00000020:00200000:3.0:1789760686.449757:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760686.449767:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760686.449801:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19439616, count: 4096 00000080:00200000:3.0:1789760686.449809:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19439616, bytes: 4096 00000080:00200000:3.0:1789760686.449820:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760686.449828:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760686.449831:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19439616, 19443712) 0 0 00000080:00200000:3.0:1789760686.449835:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760686.449852:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4746, 4746] 00020000:00200000:3.0:1789760686.449860:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19439616 chunk: [19439616, 19443712) 19443712, 4096 00020000:00200000:3.0:1789760686.449865:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760686.449886:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4746, 4746] 00000008:00010000:3.0:1789760686.449905:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760686.449911:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760686.449924:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19439616, 19443712) append 0 append lockpos 0 00000080:00200000:3.0:1789760686.449939:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19439616=0x128a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760686.450012:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760686.450015:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19443712 00000020:00010000:3.0:1789760686.450032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760686.450036:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760686.450040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760686.450043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760686.450047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760686.450055:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760686.450061:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760686.450077:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760686.450247:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 19439616<>19443712 00000080:00200000:3.0:1789760686.552952:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4746, 4746] 00000080:00200000:3.0:1789760686.552969:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760686.552975:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760686.552991:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19443712 i_blocks 38919 00000020:00200000:3.0:1789760686.552994:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760686.553015:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760686.553059:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19443712, count: 4096 00000080:00200000:3.0:1789760686.553066:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19443712, bytes: 4096 00000080:00200000:3.0:1789760686.553077:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760686.553085:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760686.553087:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19443712, 19447808) 0 0 00000080:00200000:3.0:1789760686.553091:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760686.553108:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4747, 4747] 00020000:00200000:3.0:1789760686.553116:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19443712 chunk: [19443712, 19447808) 19447808, 4096 00020000:00200000:3.0:1789760686.553120:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760686.553139:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4747, 4747] 00000008:00010000:3.0:1789760686.553158:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760686.553164:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760686.553207:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19443712, 19447808) append 0 append lockpos 0 00000080:00200000:3.0:1789760686.553223:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19443712=0x128b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760686.553280:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760686.553283:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19447808 00000020:00010000:3.0:1789760686.553291:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760686.553295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760686.553298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760686.553302:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760686.553305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760686.553314:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760686.553321:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760686.553467:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760686.553605:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 19443712<>19447808 00000100:00100000:3.0:1789760686.553657:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760686.553676:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495683712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.641602:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972064 gen 2 00000008:00010000:3.0:1789760686.641661:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760686.641670:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760686.641681:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760686.641763:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760686.641806:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495683712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760686.642637:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495683840:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760687.478134:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760687.478212:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495686528:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760687.544473:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972083 gen 2 00000008:00010000:3.0:1789760687.544525:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760687.544533:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760687.544543:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760687.544634:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760687.544665:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495686528:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760687.546597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495686656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760687.581616:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972084 gen 2 00000008:00010000:3.0:1789760687.581673:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760687.581683:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760687.581695:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760687.581720:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760687.581752:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495686656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.116826:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495687808:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760688.140434:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760688.140449:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.140461:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.140509:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760688.140599:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495687808:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.141118:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760688.141132:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495687936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.161837:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972094 gen 2 00000008:00010000:3.0:1789760688.161932:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760688.161939:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.161947:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.161967:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760688.162064:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495687936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.221989:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.251834:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972097 gen 2 00000008:00010000:3.0:1789760688.251885:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760688.251894:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.251906:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.251928:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760688.251959:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.253082:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760688.253204:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688448:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.318908:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972098 gen 2 00000008:00010000:3.0:1789760688.318969:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760688.318978:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.319003:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.319038:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760688.319069:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688448:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.320822:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.363211:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972099 gen 2 00000008:00010000:3.0:1789760688.363263:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760688.363272:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.363283:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.363308:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760688.363347:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.365561:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760688.365661:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.418495:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972100 gen 2 00000008:00010000:3.0:1789760688.418547:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760688.418555:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.418568:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.418595:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760688.418623:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.419713:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.503245:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972101 gen 2 00000008:00010000:3.0:1789760688.503299:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760688.503307:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.503321:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.503354:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760688.503384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495688832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.653941:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495689216:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.725618:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972104 gen 2 00000008:00010000:3.0:1789760688.725685:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760688.725694:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.725705:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.725732:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760688.725761:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495689216:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.726628:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760688.726645:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495689344:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.788485:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972105 gen 2 00000008:00010000:3.0:1789760688.788537:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760688.788547:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760688.788560:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760688.788579:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760688.788611:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495689344:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760688.789776:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495689472:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760689.019713:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4792, 4792] 00000080:00200000:3.0:1789760689.019734:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760689.019740:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760689.019758:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19632128 i_blocks 38919 00000020:00200000:3.0:1789760689.019762:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760689.019772:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760689.019809:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19632128, count: 4096 00000080:00200000:3.0:1789760689.019817:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19632128, bytes: 4096 00000080:00200000:3.0:1789760689.019830:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760689.019849:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760689.019851:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19632128, 19636224) 0 0 00000080:00200000:3.0:1789760689.019856:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760689.019874:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4793, 4793] 00020000:00200000:3.0:1789760689.019882:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19632128 chunk: [19632128, 19636224) 19636224, 4096 00020000:00200000:3.0:1789760689.019887:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760689.019910:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4793, 4793] 00000008:00010000:3.0:1789760689.019929:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760689.019935:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760689.019947:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19632128, 19636224) append 0 append lockpos 0 00000080:00200000:3.0:1789760689.019962:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19632128=0x12b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760689.020045:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760689.020048:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19636224 00000020:00010000:3.0:1789760689.020057:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760689.020061:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760689.020064:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760689.020067:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760689.020070:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760689.020078:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760689.020086:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760689.020103:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760689.020280:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 19632128<>19636224 00000100:00100000:3.0:1789760689.020559:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760689.020579:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495689984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.117968:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972110 gen 2 00000008:00010000:3.0:1789760689.118037:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760689.118045:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760689.118056:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760689.118079:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760689.118106:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495689984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.455358:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760689.455386:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495690880:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.500670:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972117 gen 2 00000008:00010000:3.0:1789760689.500725:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760689.500734:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760689.500746:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760689.500764:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760689.500794:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495690880:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.509449:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495691008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.573281:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972118 gen 2 00000008:00010000:3.0:1789760689.573345:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760689.573355:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760689.573367:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760689.573483:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760689.573517:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495691008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.574722:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495691136:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760689.628828:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760689.628845:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760689.628869:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760689.628894:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760689.628927:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495691136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.630188:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760689.630210:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495691264:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.671762:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972120 gen 2 00000008:00010000:3.0:1789760689.671827:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760689.671836:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760689.671858:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760689.671881:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760689.671910:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495691264:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.761805:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495691648:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.858599:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495691904:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.927746:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972125 gen 2 00000008:00010000:3.0:1789760689.927805:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760689.927815:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760689.927828:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760689.927875:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760689.927909:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495691904:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760689.931072:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760689.931106:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692032:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.003335:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972126 gen 2 00000008:00010000:3.0:1789760690.003388:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760690.003396:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.003408:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.003437:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760690.003464:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692032:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.007959:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692160:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.113302:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972127 gen 2 00000008:00010000:3.0:1789760690.113357:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760690.113366:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.113379:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.113409:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760690.113441:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692160:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.117891:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760690.117919:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692288:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760690.177400:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760690.177416:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.177428:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.177454:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760690.177484:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692288:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.179235:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.279436:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972129 gen 2 00000008:00010000:3.0:1789760690.279497:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760690.279504:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.279514:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.279536:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760690.279564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.280588:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760690.280609:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.430253:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972130 gen 2 00000008:00010000:3.0:1789760690.430305:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760690.430320:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.430332:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.430355:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760690.430387:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.431328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692672:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.505999:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972131 gen 2 00000008:00010000:3.0:1789760690.506050:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760690.506058:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.506071:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.506095:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760690.506124:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692672:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.507020:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495692800:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.552014:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972132 gen 2 00000008:00010000:3.0:1789760690.552067:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760690.552076:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.552087:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.552113:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760690.552143:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495692800:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.553260:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692928:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760690.673816:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760690.673835:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.673848:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.673877:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760690.673909:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495692928:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.675445:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495693056:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.757448:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972134 gen 2 00000008:00010000:3.0:1789760690.757513:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760690.757522:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.757532:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.757560:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760690.757588:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495693056:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.759333:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495693184:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760690.840114:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972135 gen 2 00000008:00010000:3.0:1789760690.840166:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760690.840209:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760690.840221:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760690.840249:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760690.840279:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495693184:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760690.911725:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4819, 4819] 00000080:00200000:3.0:1789760690.911756:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760690.911763:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760690.911783:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19742720 i_blocks 38919 00000020:00200000:3.0:1789760690.911789:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760690.911807:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760690.911843:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19742720, count: 4096 00000080:00200000:3.0:1789760690.911853:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19742720, bytes: 4096 00000080:00200000:3.0:1789760690.911870:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760690.911892:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760690.911895:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19742720, 19746816) 0 0 00000080:00200000:3.0:1789760690.911899:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760690.911918:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4820, 4820] 00020000:00200000:3.0:1789760690.911927:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19742720 chunk: [19742720, 19746816) 19746816, 4096 00020000:00200000:3.0:1789760690.911933:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760690.911957:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4820, 4820] 00000008:00010000:3.0:1789760690.911979:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760690.911996:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760690.912010:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19742720, 19746816) append 0 append lockpos 0 00000080:00200000:3.0:1789760690.912028:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19742720=0x12d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760690.912108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760690.912112:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19746816 00000020:00010000:3.0:1789760690.912121:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760690.912125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760690.912128:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760690.912131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760690.912134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760690.912144:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760690.912151:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760690.912222:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760690.912476:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 19742720<>19746816 00000080:00200000:3.0:1789760690.979669:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4820, 4820] 00000080:00200000:3.0:1789760690.979697:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760690.979703:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760690.979719:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19746816 i_blocks 38919 00000020:00200000:3.0:1789760690.979722:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760690.979732:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760690.979767:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19746816, count: 4096 00000080:00200000:3.0:1789760690.979775:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19746816, bytes: 4096 00000080:00200000:3.0:1789760690.979785:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760690.979794:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760690.979796:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19746816, 19750912) 0 0 00000080:00200000:3.0:1789760690.979800:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760690.979817:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4821, 4821] 00020000:00200000:3.0:1789760690.979826:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19746816 chunk: [19746816, 19750912) 19750912, 4096 00020000:00200000:3.0:1789760690.979831:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760690.979851:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4821, 4821] 00000008:00010000:3.0:1789760690.979870:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760690.979876:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760690.979887:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19746816, 19750912) append 0 append lockpos 0 00000080:00200000:3.0:1789760690.979902:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19746816=0x12d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760690.979957:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760690.979960:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19750912 00000020:00010000:3.0:1789760690.979968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760690.979972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760690.979975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760690.979978:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760690.979981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760690.980001:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760690.980009:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760690.980026:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760690.981264:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 19746816<>19750912 00000100:00100000:3.0:1789760691.183577:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495693952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.209815:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972141 gen 2 00000008:00010000:3.0:1789760691.209939:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760691.209949:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.209959:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.210084:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760691.210114:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495693952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.211209:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495694080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.255791:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972142 gen 2 00000008:00010000:3.0:1789760691.256394:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760691.256402:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.256415:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.256435:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760691.256464:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495694080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.263522:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760691.263569:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495694208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.323445:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972143 gen 2 00000008:00010000:3.0:1789760691.323495:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760691.323514:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.323527:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.323554:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760691.323584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495694208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.427515:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495694592:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760691.468534:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760691.468550:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.468561:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.468588:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760691.468619:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495694592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.469615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495694720:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.529305:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972147 gen 2 00000008:00010000:3.0:1789760691.529879:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760691.529889:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.529900:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.530324:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760691.530368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495694720:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760691.630361:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4831, 4831] 00000080:00200000:3.0:1789760691.630377:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760691.630383:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760691.630398:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19791872 i_blocks 38919 00000020:00200000:3.0:1789760691.630402:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760691.630411:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760691.630444:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19791872, count: 4096 00000080:00200000:3.0:1789760691.630451:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19791872, bytes: 4096 00000080:00200000:3.0:1789760691.630460:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760691.630468:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760691.630471:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19791872, 19795968) 0 0 00000080:00200000:3.0:1789760691.630475:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760691.630492:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4832, 4832] 00020000:00200000:3.0:1789760691.630500:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19791872 chunk: [19791872, 19795968) 19795968, 4096 00020000:00200000:3.0:1789760691.630505:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760691.630523:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4832, 4832] 00000008:00010000:3.0:1789760691.630539:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760691.630553:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760691.630565:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19791872, 19795968) append 0 append lockpos 0 00000080:00200000:3.0:1789760691.630579:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19791872=0x12e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760691.630781:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760691.630784:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19795968 00000020:00010000:3.0:1789760691.630793:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760691.630796:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760691.630800:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760691.630803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760691.630806:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760691.630813:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760691.630819:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760691.630835:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760691.631009:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 19791872<>19795968 00000100:00100000:3.0:1789760691.631057:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760691.631080:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495694976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760691.666809:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972149 gen 2 00000008:00010000:3.0:1789760691.666925:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760691.666934:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.666945:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.667091:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760691.667122:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495694976:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760691.715926:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4833, 4833] 00000080:00200000:3.0:1789760691.715945:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760691.715952:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760691.716373:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19800064 i_blocks 38919 00000020:00200000:3.0:1789760691.716381:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760691.716392:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760691.716426:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19800064, count: 4096 00000080:00200000:3.0:1789760691.716434:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19800064, bytes: 4096 00000080:00200000:3.0:1789760691.716448:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760691.716456:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760691.716459:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19800064, 19804160) 0 0 00000080:00200000:3.0:1789760691.716463:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760691.716492:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4834, 4834] 00020000:00200000:3.0:1789760691.716500:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19800064 chunk: [19800064, 19804160) 19804160, 4096 00020000:00200000:3.0:1789760691.716505:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760691.716526:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4834, 4834] 00000008:00010000:3.0:1789760691.716545:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760691.716550:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760691.716562:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19800064, 19804160) append 0 append lockpos 0 00000080:00200000:3.0:1789760691.716576:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19800064=0x12e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760691.716644:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760691.716647:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19804160 00000020:00010000:3.0:1789760691.716656:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760691.716660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760691.716664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760691.716669:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760691.716672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760691.716680:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760691.716686:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760691.716710:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760691.716837:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 19800064<>19804160 00000100:00100000:3.0:1789760691.749012:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972151 gen 2 00000008:00010000:3.0:1789760691.749089:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760691.749098:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.749109:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.749132:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760691.749165:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495695232:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.196765:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495696768:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.223751:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972160 gen 2 00000008:00010000:3.0:1789760692.223802:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760692.223810:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.223822:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.223848:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760692.223877:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495696768:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.224913:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760692.224932:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495696896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.253428:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972161 gen 2 00000008:00010000:3.0:1789760692.253487:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760692.253495:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.253507:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.253530:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760692.253559:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495696896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.254446:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495697024:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.297323:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972162 gen 2 00000008:00010000:3.0:1789760692.297372:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760692.297382:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.297394:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.297421:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760692.297451:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495697024:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.298055:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495697152:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760692.336435:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760692.336453:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.336466:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.336494:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760692.336525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495697152:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.337620:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495697280:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.408796:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972163 gen 2 00000100:00100000:3.0:1789760692.408843:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264e00 x1876699495697280/t4294972164(4294972164) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760708 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760692.408908:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760692.408917:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.408936:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.408961:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760692.408995:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495697280:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.410341:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495697408:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.544965:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972165 gen 2 00000008:00010000:3.0:1789760692.545041:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760692.545051:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.545061:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.545087:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760692.545118:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495697408:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.552524:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495697536:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.608161:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972166 gen 2 00000008:00010000:3.0:1789760692.608249:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760692.608259:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.608271:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.608305:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760692.608336:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495697536:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760692.609103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495697664:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760692.670606:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4850, 4850] 00000080:00200000:3.0:1789760692.670626:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760692.670633:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760692.670650:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19869696 i_blocks 38919 00000020:00200000:3.0:1789760692.670653:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760692.670663:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760692.670699:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19869696, count: 4096 00000080:00200000:3.0:1789760692.670706:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19869696, bytes: 4096 00000080:00200000:3.0:1789760692.670717:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760692.670727:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760692.670729:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19869696, 19873792) 0 0 00000080:00200000:3.0:1789760692.670734:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760692.670751:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4851, 4851] 00020000:00200000:3.0:1789760692.670760:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19869696 chunk: [19869696, 19873792) 19873792, 4096 00020000:00200000:3.0:1789760692.670765:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760692.670793:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4851, 4851] 00000008:00010000:3.0:1789760692.670815:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760692.670820:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760692.670832:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19869696, 19873792) append 0 append lockpos 0 00000080:00200000:3.0:1789760692.670849:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19869696=0x12f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760692.670928:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760692.670931:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19873792 00000020:00010000:3.0:1789760692.670941:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760692.670945:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760692.670948:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760692.670952:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760692.670955:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760692.670963:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760692.670970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760692.670996:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760692.671125:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 19869696<>19873792 00000080:00200000:3.0:1789760692.810161:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4851, 4851] 00000080:00200000:3.0:1789760692.810193:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760692.810200:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760692.810216:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19873792 i_blocks 38919 00000020:00200000:3.0:1789760692.810220:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760692.810230:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760692.810264:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19873792, count: 4096 00000080:00200000:3.0:1789760692.810271:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19873792, bytes: 4096 00000080:00200000:3.0:1789760692.810285:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760692.810294:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760692.810296:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19873792, 19877888) 0 0 00000080:00200000:3.0:1789760692.810301:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760692.810321:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4852, 4852] 00020000:00200000:3.0:1789760692.810329:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19873792 chunk: [19873792, 19877888) 19877888, 4096 00020000:00200000:3.0:1789760692.810334:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760692.810356:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4852, 4852] 00000008:00010000:3.0:1789760692.810374:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760692.810380:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760692.810392:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19873792, 19877888) append 0 append lockpos 0 00000080:00200000:3.0:1789760692.810405:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19873792=0x12f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760692.810537:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760692.810540:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19877888 00000020:00010000:3.0:1789760692.810550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760692.810554:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760692.810557:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760692.810560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760692.810563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760692.810572:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760692.810580:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760692.810596:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760692.810860:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 19873792<>19877888 00000080:00200000:3.0:1789760693.010278:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4852, 4852] 00000080:00200000:3.0:1789760693.010308:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760693.010315:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760693.010382:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19877888 i_blocks 38919 00000020:00200000:3.0:1789760693.010388:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760693.010405:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760693.010440:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19877888, count: 4096 00000080:00200000:3.0:1789760693.010449:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19877888, bytes: 4096 00000080:00200000:3.0:1789760693.010464:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760693.010474:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760693.010476:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19877888, 19881984) 0 0 00000080:00200000:3.0:1789760693.010481:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760693.010498:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4853, 4853] 00020000:00200000:3.0:1789760693.010508:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19877888 chunk: [19877888, 19881984) 19881984, 4096 00020000:00200000:3.0:1789760693.010513:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760693.010535:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4853, 4853] 00000008:00010000:3.0:1789760693.010554:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760693.010559:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760693.010572:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19877888, 19881984) append 0 append lockpos 0 00000080:00200000:3.0:1789760693.010591:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19877888=0x12f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760693.010656:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760693.010659:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19881984 00000020:00010000:3.0:1789760693.010668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760693.010671:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.010675:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.010679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760693.010682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760693.010690:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760693.010698:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760693.010716:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760693.010849:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 19877888<>19881984 00000080:00200000:3.0:1789760693.148838:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4853, 4853] 00000080:00200000:3.0:1789760693.148855:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760693.148861:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760693.148875:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19881984 i_blocks 38919 00000020:00200000:3.0:1789760693.148879:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760693.148888:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760693.148921:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19881984, count: 4096 00000080:00200000:3.0:1789760693.148928:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19881984, bytes: 4096 00000080:00200000:3.0:1789760693.148937:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760693.148946:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760693.148948:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19881984, 19886080) 0 0 00000080:00200000:3.0:1789760693.148952:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760693.148979:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4854, 4854] 00020000:00200000:3.0:1789760693.148987:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19881984 chunk: [19881984, 19886080) 19886080, 4096 00020000:00200000:3.0:1789760693.148992:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760693.149011:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4854, 4854] 00000008:00010000:3.0:1789760693.149030:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760693.149035:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760693.149047:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19881984, 19886080) append 0 append lockpos 0 00000080:00200000:3.0:1789760693.149062:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19881984=0x12f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760693.149118:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760693.149121:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19886080 00000020:00010000:3.0:1789760693.149129:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760693.149132:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.149135:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.149138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760693.149141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760693.149149:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760693.149156:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760693.149219:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760693.149342:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 19881984<>19886080 00000100:00100000:3.0:1789760693.149388:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760693.149407:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495698176:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760693.190162:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972171 gen 2 00000008:00010000:3.0:1789760693.190249:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760693.190258:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.190269:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.190296:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760693.190325:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495698176:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760693.191058:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495698304:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760693.259090:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972172 gen 2 00000008:00010000:3.0:1789760693.259141:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760693.259150:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.259162:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.259218:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760693.259247:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495698304:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760693.328321:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4856, 4856] 00000080:00200000:3.0:1789760693.328339:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760693.328345:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760693.328360:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19894272 i_blocks 38919 00000020:00200000:3.0:1789760693.328364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760693.328373:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760693.328407:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19894272, count: 4096 00000080:00200000:3.0:1789760693.328415:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19894272, bytes: 4096 00000080:00200000:3.0:1789760693.328427:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760693.328435:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760693.328438:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19894272, 19898368) 0 0 00000080:00200000:3.0:1789760693.328443:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760693.328459:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4857, 4857] 00020000:00200000:3.0:1789760693.328468:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19894272 chunk: [19894272, 19898368) 19898368, 4096 00020000:00200000:3.0:1789760693.328473:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760693.328494:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4857, 4857] 00000008:00010000:3.0:1789760693.328514:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760693.328520:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760693.328532:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19894272, 19898368) append 0 append lockpos 0 00000080:00200000:3.0:1789760693.328547:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19894272=0x12f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760693.328610:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760693.328613:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19898368 00000020:00010000:3.0:1789760693.328753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760693.328758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.328762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.328765:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760693.328768:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760693.328778:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760693.328786:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760693.328805:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760693.328959:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 19894272<>19898368 00000080:00200000:3.0:1789760693.439056:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4857, 4857] 00000080:00200000:3.0:1789760693.439076:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760693.439082:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760693.439098:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19898368 i_blocks 38919 00000020:00200000:3.0:1789760693.439102:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760693.439112:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760693.439148:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19898368, count: 4096 00000080:00200000:3.0:1789760693.439156:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19898368, bytes: 4096 00000080:00200000:3.0:1789760693.439189:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760693.439199:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760693.439202:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19898368, 19902464) 0 0 00000080:00200000:3.0:1789760693.439207:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760693.439224:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4858, 4858] 00020000:00200000:3.0:1789760693.439233:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19898368 chunk: [19898368, 19902464) 19902464, 4096 00020000:00200000:3.0:1789760693.439238:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760693.439258:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4858, 4858] 00000008:00010000:3.0:1789760693.439275:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760693.439281:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760693.439407:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19898368, 19902464) append 0 append lockpos 0 00000080:00200000:3.0:1789760693.439421:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19898368=0x12fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760693.439477:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760693.439480:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19902464 00000020:00010000:3.0:1789760693.439488:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760693.439491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.439495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.439499:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760693.439502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760693.439510:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760693.439517:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760693.439534:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760693.439664:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 19898368<>19902464 00000100:00100000:3.0:1789760693.439785:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760693.439804:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495698688:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760693.523123:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972175 gen 2 00000008:00010000:3.0:1789760693.523233:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:3.0:1789760693.523242:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.523252:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.523271:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:3.0:1789760693.523306:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495698688:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760693.559022:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4859, 4859] 00000080:00200000:3.0:1789760693.559043:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760693.559051:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760694.108709:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495700096:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760694.162393:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760694.162410:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.162423:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.162452:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760694.162485:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495700096:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760694.210122:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4870, 4870] 00000080:00200000:3.0:1789760694.210138:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.210144:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.210160:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19951616 i_blocks 40967 00000020:00200000:3.0:1789760694.210163:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.210201:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.210269:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19951616, count: 4096 00000080:00200000:3.0:1789760694.210276:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19951616, bytes: 4096 00000080:00200000:3.0:1789760694.210295:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.210303:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.210306:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19951616, 19955712) 0 0 00000080:00200000:3.0:1789760694.210310:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.210326:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4871, 4871] 00020000:00200000:3.0:1789760694.210334:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19951616 chunk: [19951616, 19955712) 19955712, 4096 00020000:00200000:3.0:1789760694.210339:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.210359:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4871, 4871] 00000008:00010000:3.0:1789760694.210375:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.210380:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.210390:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19951616, 19955712) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.210404:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19951616=0x1307000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.210451:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.210454:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19955712 00000020:00010000:3.0:1789760694.210462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.210465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.210468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.210471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.210474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.210480:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.210486:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.210500:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760694.210610:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 19951616<>19955712 00000100:00100000:3.0:1789760694.210656:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760694.210671:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495700352:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760694.248602:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:3.0:1789760694.248618:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.248628:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.248655:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:3.0:1789760694.248685:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495700352:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760694.298640:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4872, 4872] 00000080:00200000:3.0:1789760694.298658:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.298665:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.298683:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19959808 i_blocks 40967 00000020:00200000:3.0:1789760694.298687:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.298697:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.298728:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19959808, count: 4096 00000080:00200000:3.0:1789760694.298734:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19959808, bytes: 4096 00000080:00200000:3.0:1789760694.298747:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.298756:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.298758:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19959808, 19963904) 0 0 00000080:00200000:3.0:1789760694.298762:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.298779:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4873, 4873] 00020000:00200000:3.0:1789760694.298788:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19959808 chunk: [19959808, 19963904) 19963904, 4096 00020000:00200000:3.0:1789760694.298793:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.298814:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4873, 4873] 00000008:00010000:3.0:1789760694.298833:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.298838:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.298894:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19959808, 19963904) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.298910:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19959808=0x1309000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.299132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.299135:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19963904 00000020:00010000:3.0:1789760694.299145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.299149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.299152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.299155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.299158:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.299166:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.299269:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.299359:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760694.299490:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 19959808<>19963904 00000080:00200000:3.0:1789760694.342591:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4873, 4873] 00000080:00200000:3.0:1789760694.342617:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.342625:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.342642:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19963904 i_blocks 40967 00000020:00200000:3.0:1789760694.342649:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.342664:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.342697:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19963904, count: 4096 00000080:00200000:3.0:1789760694.342706:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19963904, bytes: 4096 00000080:00200000:3.0:1789760694.342721:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.342730:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.342732:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19963904, 19968000) 0 0 00000080:00200000:3.0:1789760694.342737:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.342753:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4874, 4874] 00020000:00200000:3.0:1789760694.342762:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19963904 chunk: [19963904, 19968000) 19968000, 4096 00020000:00200000:3.0:1789760694.342767:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.342789:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4874, 4874] 00000008:00010000:3.0:1789760694.342808:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.342814:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.342826:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19963904, 19968000) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.342842:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19963904=0x130a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.342914:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.342917:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19968000 00000020:00010000:3.0:1789760694.342942:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.342946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.342949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.342953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.342955:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.342971:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.342978:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.343009:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760694.343413:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 19963904<>19968000 00000080:00200000:3.0:1789760694.424750:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4874, 4874] 00000080:00200000:3.0:1789760694.424767:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.424774:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.424789:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19968000 i_blocks 40967 00000020:00200000:3.0:1789760694.424793:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.424802:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.424837:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19968000, count: 4096 00000080:00200000:3.0:1789760694.424844:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19968000, bytes: 4096 00000080:00200000:3.0:1789760694.424865:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.424875:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.424878:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19968000, 19972096) 0 0 00000080:00200000:3.0:1789760694.424883:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.424900:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4875, 4875] 00020000:00200000:3.0:1789760694.424909:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19968000 chunk: [19968000, 19972096) 19972096, 4096 00020000:00200000:3.0:1789760694.424914:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.424934:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4875, 4875] 00000008:00010000:3.0:1789760694.424968:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.424975:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.424987:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19968000, 19972096) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.425002:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19968000=0x130b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.425066:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.425069:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19972096 00000020:00010000:3.0:1789760694.425077:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.425081:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.425085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.425088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.425091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.425101:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.425108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.425127:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760694.425481:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 19968000<>19972096 00000080:00200000:3.0:1789760694.485976:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4875, 4875] 00000080:00200000:3.0:1789760694.485993:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.486002:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.486018:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19972096 i_blocks 40967 00000020:00200000:3.0:1789760694.486736:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.486750:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.486786:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19972096, count: 4096 00000080:00200000:3.0:1789760694.486794:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19972096, bytes: 4096 00000080:00200000:3.0:1789760694.486807:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.486816:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.486818:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19972096, 19976192) 0 0 00000080:00200000:3.0:1789760694.486824:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.486842:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4876, 4876] 00020000:00200000:3.0:1789760694.486860:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19972096 chunk: [19972096, 19976192) 19976192, 4096 00020000:00200000:3.0:1789760694.486865:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.486886:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4876, 4876] 00000008:00010000:3.0:1789760694.486905:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.486911:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.487254:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19972096, 19976192) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.487270:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19972096=0x130c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.487398:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.487401:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19976192 00000020:00010000:3.0:1789760694.487410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.487414:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.487417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.487420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.487423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.487432:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.487439:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.487464:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760694.487589:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 19972096<>19976192 00000080:00200000:3.0:1789760694.526649:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4876, 4876] 00000080:00200000:3.0:1789760694.526666:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.526671:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.571502:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4877, 4877] 00000080:00200000:3.0:1789760694.571520:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.571526:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.571542:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19980288 i_blocks 40967 00000020:00200000:3.0:1789760694.571546:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.571555:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.571619:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19980288, count: 4096 00000080:00200000:3.0:1789760694.571627:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19980288, bytes: 4096 00000080:00200000:3.0:1789760694.571638:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.571648:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.571650:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19980288, 19984384) 0 0 00000080:00200000:3.0:1789760694.571655:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.571683:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4878, 4878] 00020000:00200000:3.0:1789760694.571692:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19980288 chunk: [19980288, 19984384) 19984384, 4096 00020000:00200000:3.0:1789760694.571696:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.571719:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4878, 4878] 00000008:00010000:3.0:1789760694.571737:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.571748:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.571760:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19980288, 19984384) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.571774:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19980288=0x130e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.571833:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.571836:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19984384 00000020:00010000:3.0:1789760694.571845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.571848:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.571852:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.571855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.571859:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.571866:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.571873:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.571889:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760694.572705:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 19980288<>19984384 00000100:00100000:3.0:1789760694.572765:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760694.572785:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495701248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760694.620015:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972195 gen 2 00000008:00010000:3.0:1789760694.620077:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:3.0:1789760694.620085:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.620095:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.620120:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:3.0:1789760694.620158:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495701248:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760694.689233:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4879, 4879] 00000080:00200000:3.0:1789760694.689250:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.689257:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.689272:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19988480 i_blocks 40967 00000020:00200000:3.0:1789760694.689276:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.689295:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.689330:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19988480, count: 4096 00000080:00200000:3.0:1789760694.689338:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19988480, bytes: 4096 00000080:00200000:3.0:1789760694.689348:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.689357:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.689360:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19988480, 19992576) 0 0 00000080:00200000:3.0:1789760694.689364:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.689381:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4880, 4880] 00020000:00200000:3.0:1789760694.689390:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19988480 chunk: [19988480, 19992576) 19992576, 4096 00020000:00200000:3.0:1789760694.689395:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.689414:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4880, 4880] 00000008:00010000:3.0:1789760694.689432:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.689438:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.689451:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19988480, 19992576) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.689466:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19988480=0x1310000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.689522:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.689526:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19992576 00000020:00010000:3.0:1789760694.689535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.689538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.689541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.689544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.689547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.689554:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.689561:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.689579:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760694.689709:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 19988480<>19992576 00000080:00200000:3.0:1789760694.755413:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4880, 4880] 00000080:00200000:3.0:1789760694.755431:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.755437:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.755453:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19992576 i_blocks 40967 00000020:00200000:3.0:1789760694.755458:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.755467:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.755501:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19992576, count: 4096 00000080:00200000:3.0:1789760694.755509:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19992576, bytes: 4096 00000080:00200000:3.0:1789760694.755521:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.755529:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.755531:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19992576, 19996672) 0 0 00000080:00200000:3.0:1789760694.755536:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.755554:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4881, 4881] 00020000:00200000:3.0:1789760694.755562:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19992576 chunk: [19992576, 19996672) 19996672, 4096 00020000:00200000:3.0:1789760694.755566:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.755587:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4881, 4881] 00000008:00010000:3.0:1789760694.755605:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.755612:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.755624:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19992576, 19996672) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.755640:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19992576=0x1311000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.755704:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.755707:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 19996672 00000020:00010000:3.0:1789760694.755716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.755719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.755722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.755726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.755729:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.755736:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.755744:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.755762:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760694.755890:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 19992576<>19996672 00000080:00200000:3.0:1789760694.828003:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4881, 4881] 00000080:00200000:3.0:1789760694.828020:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.828027:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.828043:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 19996672 i_blocks 40967 00000020:00200000:3.0:1789760694.828047:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.828057:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.828093:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 19996672, count: 4096 00000080:00200000:3.0:1789760694.828100:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 19996672, bytes: 4096 00000080:00200000:3.0:1789760694.828110:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.828118:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.828121:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [19996672, 20000768) 0 0 00000080:00200000:3.0:1789760694.828126:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.828144:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4882, 4882] 00020000:00200000:3.0:1789760694.828153:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 19996672 chunk: [19996672, 20000768) 20000768, 4096 00020000:00200000:3.0:1789760694.828158:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.828211:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4882, 4882] 00000008:00010000:3.0:1789760694.828231:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.828237:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.828250:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [19996672, 20000768) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.828264:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=19996672=0x1312000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.828330:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.828333:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20000768 00000020:00010000:3.0:1789760694.828342:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.828346:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.828349:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.828352:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.828355:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.828362:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.828368:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.828384:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760694.828515:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 19996672<>20000768 00000080:00200000:3.0:1789760694.940664:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4882, 4882] 00000080:00200000:3.0:1789760694.940683:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.940689:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.940705:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20000768 i_blocks 40967 00000020:00200000:3.0:1789760694.940709:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.940719:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.940755:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20000768, count: 4096 00000080:00200000:3.0:1789760694.940762:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20000768, bytes: 4096 00000080:00200000:3.0:1789760694.940783:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.940794:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.940796:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20000768, 20004864) 0 0 00000080:00200000:3.0:1789760694.940801:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.940818:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4883, 4883] 00020000:00200000:3.0:1789760694.940826:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20000768 chunk: [20000768, 20004864) 20004864, 4096 00020000:00200000:3.0:1789760694.940831:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.940852:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4883, 4883] 00000008:00010000:3.0:1789760694.940870:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.940877:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760694.940888:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20000768, 20004864) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.940901:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20000768=0x1313000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.940974:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.940977:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20004864 00000020:00010000:3.0:1789760694.940985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.940988:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.940991:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.940995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.940998:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.941006:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.941014:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.941031:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760694.941156:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 20000768<>20004864 00000080:00200000:3.0:1789760695.025019:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4883, 4883] 00000080:00200000:3.0:1789760695.025044:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.025050:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.025065:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20004864 i_blocks 40967 00000020:00200000:3.0:1789760695.025068:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.025077:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.025111:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20004864, count: 4096 00000080:00200000:3.0:1789760695.025119:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20004864, bytes: 4096 00000080:00200000:3.0:1789760695.025129:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.025139:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.025141:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20004864, 20008960) 0 0 00000080:00200000:3.0:1789760695.025146:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.025164:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4884, 4884] 00020000:00200000:3.0:1789760695.025225:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20004864 chunk: [20004864, 20008960) 20008960, 4096 00020000:00200000:3.0:1789760695.025231:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.025285:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4884, 4884] 00000008:00010000:3.0:1789760695.025303:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.025310:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760695.025320:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20004864, 20008960) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.025337:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20004864=0x1314000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.025391:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.025394:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20008960 00000020:00010000:3.0:1789760695.025402:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.025406:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.025409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.025412:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.025416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.025423:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.025431:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.025443:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760695.025568:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 20004864<>20008960 00000080:00200000:3.0:1789760695.101990:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4884, 4884] 00000080:00200000:3.0:1789760695.102006:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.102012:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.102028:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20008960 i_blocks 40967 00000020:00200000:3.0:1789760695.102032:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.102042:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.102074:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20008960, count: 4096 00000080:00200000:3.0:1789760695.102081:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20008960, bytes: 4096 00000080:00200000:3.0:1789760695.102090:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.102098:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.102100:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20008960, 20013056) 0 0 00000080:00200000:3.0:1789760695.102104:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.102121:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4885, 4885] 00020000:00200000:3.0:1789760695.102129:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20008960 chunk: [20008960, 20013056) 20013056, 4096 00020000:00200000:3.0:1789760695.102134:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.102154:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4885, 4885] 00000008:00010000:3.0:1789760695.102203:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.102210:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760695.102222:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20008960, 20013056) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.102236:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20008960=0x1315000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.102301:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.102304:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20013056 00000020:00010000:3.0:1789760695.102312:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.102316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.102319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.102322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.102325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.102332:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.102338:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.102353:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760695.102471:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 20008960<>20013056 00000080:00200000:3.0:1789760695.193475:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4885, 4885] 00000080:00200000:3.0:1789760695.193492:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.193498:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.193514:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20013056 i_blocks 40967 00000020:00200000:3.0:1789760695.193518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.193527:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.193563:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20013056, count: 4096 00000080:00200000:3.0:1789760695.193571:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20013056, bytes: 4096 00000080:00200000:3.0:1789760695.193582:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.193591:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.193593:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20013056, 20017152) 0 0 00000080:00200000:3.0:1789760695.193598:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.193617:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4886, 4886] 00020000:00200000:3.0:1789760695.193625:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20013056 chunk: [20013056, 20017152) 20017152, 4096 00020000:00200000:3.0:1789760695.193630:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.193649:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4886, 4886] 00000008:00010000:3.0:1789760695.193670:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.193676:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000080:00200000:3.0:1789760695.193689:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20013056, 20017152) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.193703:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20013056=0x1316000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.193772:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.193775:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20017152 00000020:00010000:3.0:1789760695.193783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.193787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.193790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac6b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.193793:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.193796:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac6b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.193804:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.193810:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.193826:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760695.194219:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 20013056<>20017152 00000080:00200000:3.0:1789760695.294997:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4886, 4886] 00000080:00200000:3.0:1789760695.295013:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.295019:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.295034:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20017152 i_blocks 40967 00000020:00200000:3.0:1789760695.295038:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.295047:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.295079:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20017152, count: 4096 00000080:00200000:3.0:1789760695.295087:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20017152, bytes: 4096 00000080:00200000:3.0:1789760695.295096:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.295104:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.295106:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20017152, 20021248) 0 0 00000080:00200000:3.0:1789760695.295111:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.295138:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4887, 4887] 00020000:00200000:3.0:1789760695.295146:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20017152 chunk: [20017152, 20021248) 20021248, 4096 00020000:00200000:3.0:1789760695.295150:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.295203:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4887, 4887] 00000008:00010000:3.0:1789760695.295231:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.295238:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac6b0, osc lock ffff8dd2050530a0, flags 0x0 00000100:00100000:3.0:1789760695.444062:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760695.444082:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495702656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760695.536470:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972206 gen 2 00000008:00010000:3.0:1789760695.536519:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760695.536527:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.536538:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.536562:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760695.536592:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495702656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760695.537573:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760695.537591:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495702784:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760695.581911:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972207 gen 2 00000008:00010000:3.0:1789760695.582219:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760695.582228:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.582240:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.582264:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760695.582355:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495702784:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760695.583351:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760695.583434:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495702912:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760695.643114:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760695.643132:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.643146:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.643213:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760695.643244:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495702912:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760695.722572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495703296:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760695.746757:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760695.746774:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.746785:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.746811:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760695.746839:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495703296:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760695.747542:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495703424:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760695.772795:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972212 gen 2 00000080:00200000:3.0:1789760695.885150:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4897, 4897] 00000080:00200000:3.0:1789760695.885204:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.885211:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.885228:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20062208 i_blocks 40967 00000020:00200000:3.0:1789760695.885231:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.885243:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.885285:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20062208, count: 4096 00000080:00200000:3.0:1789760695.885292:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20062208, bytes: 4096 00000080:00200000:3.0:1789760695.885303:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.885312:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.885315:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20062208, 20066304) 0 0 00000080:00200000:3.0:1789760695.885319:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.885337:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4898, 4898] 00020000:00200000:3.0:1789760695.885345:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20062208 chunk: [20062208, 20066304) 20066304, 4096 00020000:00200000:3.0:1789760695.885350:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.885371:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4898, 4898] 00000008:00010000:3.0:1789760695.885390:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.885397:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760695.885409:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20062208, 20066304) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.885423:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20062208=0x1322000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.885587:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.885591:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20066304 00000020:00010000:3.0:1789760695.885600:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.885673:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.885677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.885681:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.885684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.885695:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.885703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.885721:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760695.885869:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 20062208<>20066304 00000100:00100000:3.0:1789760695.885911:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760695.886378:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495703808:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760696.091748:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4900, 4900] 00000080:00200000:3.0:1789760696.091765:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760696.091771:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760696.091787:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20074496 i_blocks 40967 00000020:00200000:3.0:1789760696.091791:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760696.091802:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760696.091837:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20074496, count: 4096 00000080:00200000:3.0:1789760696.091844:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20074496, bytes: 4096 00000080:00200000:3.0:1789760696.091856:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760696.091865:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760696.091867:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20074496, 20078592) 0 0 00000080:00200000:3.0:1789760696.091871:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760696.091902:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4901, 4901] 00020000:00200000:3.0:1789760696.091910:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20074496 chunk: [20074496, 20078592) 20078592, 4096 00020000:00200000:3.0:1789760696.091916:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760696.092101:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4901, 4901] 00000008:00010000:3.0:1789760696.092119:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760696.092125:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760696.092137:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20074496, 20078592) append 0 append lockpos 0 00000080:00200000:3.0:1789760696.092151:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20074496=0x1325000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760696.092240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760696.092244:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20078592 00000020:00010000:3.0:1789760696.092252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760696.092256:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.092259:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.092262:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760696.092277:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760696.092284:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760696.092291:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760696.092308:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760696.092431:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 20074496<>20078592 00000080:00200000:3.0:1789760696.144390:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4901, 4901] 00000080:00200000:3.0:1789760696.144409:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760696.144415:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760696.144430:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20078592 i_blocks 40967 00000020:00200000:3.0:1789760696.144434:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760696.144444:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760696.144475:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20078592, count: 4096 00000080:00200000:3.0:1789760696.144483:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20078592, bytes: 4096 00000080:00200000:3.0:1789760696.144494:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760696.144502:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760696.144505:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20078592, 20082688) 0 0 00000080:00200000:3.0:1789760696.144510:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760696.144526:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4902, 4902] 00020000:00200000:3.0:1789760696.144535:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20078592 chunk: [20078592, 20082688) 20082688, 4096 00020000:00200000:3.0:1789760696.144539:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760696.144559:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4902, 4902] 00000008:00010000:3.0:1789760696.144584:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760696.144591:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760696.144603:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20078592, 20082688) append 0 append lockpos 0 00000080:00200000:3.0:1789760696.144619:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20078592=0x1326000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760696.145222:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760696.145227:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20082688 00000020:00010000:3.0:1789760696.145237:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760696.145241:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.145245:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.145249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760696.145252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760696.145261:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760696.145338:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760696.145357:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760696.145480:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 20078592<>20082688 00000080:00200000:3.0:1789760696.180469:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4902, 4902] 00000080:00200000:3.0:1789760696.180485:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760696.180491:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760696.180504:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20082688 i_blocks 40967 00000020:00200000:3.0:1789760696.180508:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760696.180516:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760696.180547:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20082688, count: 4096 00000080:00200000:3.0:1789760696.180553:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20082688, bytes: 4096 00000080:00200000:3.0:1789760696.180562:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760696.180570:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760696.180573:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20082688, 20086784) 0 0 00000080:00200000:3.0:1789760696.180577:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760696.180592:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4903, 4903] 00020000:00200000:3.0:1789760696.180599:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20082688 chunk: [20082688, 20086784) 20086784, 4096 00020000:00200000:3.0:1789760696.180604:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760696.180622:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4903, 4903] 00000008:00010000:3.0:1789760696.180637:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760696.180642:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760696.180653:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20082688, 20086784) append 0 append lockpos 0 00000080:00200000:3.0:1789760696.180666:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20082688=0x1327000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760696.180714:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760696.180717:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20086784 00000020:00010000:3.0:1789760696.180724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760696.180727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.180731:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.180734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760696.180737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760696.180744:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760696.180750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760696.180764:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760696.181141:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 20082688<>20086784 00000080:00200000:3.0:1789760696.241537:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4903, 4903] 00000080:00200000:3.0:1789760696.241554:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760696.241560:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760696.241576:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20086784 i_blocks 40967 00000020:00200000:3.0:1789760696.241580:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760696.241589:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760696.241624:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20086784, count: 4096 00000080:00200000:3.0:1789760696.241632:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20086784, bytes: 4096 00000080:00200000:3.0:1789760696.241642:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760696.241650:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760696.241653:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20086784, 20090880) 0 0 00000080:00200000:3.0:1789760696.241658:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760696.241676:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4904, 4904] 00020000:00200000:3.0:1789760696.241684:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20086784 chunk: [20086784, 20090880) 20090880, 4096 00020000:00200000:3.0:1789760696.241688:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760696.241709:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4904, 4904] 00000008:00010000:3.0:1789760696.241726:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760696.241732:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760696.241744:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20086784, 20090880) append 0 append lockpos 0 00000080:00200000:3.0:1789760696.241758:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20086784=0x1328000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760696.241812:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760696.241815:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20090880 00000020:00010000:3.0:1789760696.241824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760696.241827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.241830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.241833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760696.241836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760696.241844:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760696.241861:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760696.241879:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760696.242027:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 20086784<>20090880 00000080:00200000:3.0:1789760696.295110:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4904, 4904] 00000080:00200000:3.0:1789760696.295127:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760696.295134:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760696.295846:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495704704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760696.343543:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972221 gen 2 00000100:00100000:3.0:1789760696.343586:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266d80 x1876699495704704/t4294972222(4294972222) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760712 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760696.343635:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760696.343642:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.343653:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.343677:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760696.343699:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495704704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.296680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495709952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.334151:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972260 gen 2 00000008:00010000:3.0:1789760698.334232:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760698.334240:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.334310:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.334339:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760698.334374:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495709952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.335514:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760698.335530:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495710080:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760698.375636:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760698.375652:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.375664:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.375792:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760698.375825:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495710080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.376757:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495710208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.491841:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495710464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.545683:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972264 gen 2 00000008:00010000:3.0:1789760698.545745:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760698.545754:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.545765:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.545788:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760698.545826:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495710464:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760698.546773:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:3.0:1789760698.546993:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 20267008<>20271104 00000100:00100000:3.0:1789760698.547356:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760698.547376:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495710592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.789998:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495711104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.832127:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972269 gen 2 00000008:00010000:3.0:1789760698.832232:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760698.832240:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.832261:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.832283:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760698.832313:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495711104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760698.833064:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495711232:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760699.008637:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4956, 4956] 00000080:00200000:3.0:1789760699.008653:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.008660:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.008675:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20303872 i_blocks 40967 00000020:00200000:3.0:1789760699.008679:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.008687:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.008751:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20303872, count: 4096 00000080:00200000:3.0:1789760699.008759:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20303872, bytes: 4096 00000080:00200000:3.0:1789760699.008771:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.008779:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.008781:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20303872, 20307968) 0 0 00000080:00200000:3.0:1789760699.008786:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.008804:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4957, 4957] 00020000:00200000:3.0:1789760699.008813:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20303872 chunk: [20303872, 20307968) 20307968, 4096 00020000:00200000:3.0:1789760699.008817:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.008837:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4957, 4957] 00000008:00010000:3.0:1789760699.008863:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.008869:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.008881:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20303872, 20307968) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.008895:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20303872=0x135d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.009672:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.009677:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20307968 00000020:00010000:3.0:1789760699.009686:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.009690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.009693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.009697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.009700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.009708:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.009715:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.009734:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760699.009868:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 20303872<>20307968 00000080:00200000:3.0:1789760699.084935:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4957, 4957] 00000080:00200000:3.0:1789760699.084952:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.084959:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.084974:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20307968 i_blocks 40967 00000020:00200000:3.0:1789760699.084978:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.084988:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.085021:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20307968, count: 4096 00000080:00200000:3.0:1789760699.085028:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20307968, bytes: 4096 00000080:00200000:3.0:1789760699.085037:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.085047:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.085049:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20307968, 20312064) 0 0 00000080:00200000:3.0:1789760699.085054:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.085071:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4958, 4958] 00020000:00200000:3.0:1789760699.085084:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20307968 chunk: [20307968, 20312064) 20312064, 4096 00020000:00200000:3.0:1789760699.085089:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.085109:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4958, 4958] 00000008:00010000:3.0:1789760699.085128:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.085134:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.085146:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20307968, 20312064) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.085160:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20307968=0x135e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.085423:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.085427:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20312064 00000020:00010000:3.0:1789760699.085436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.085439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.085442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.085447:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.085450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.085458:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.085466:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.085484:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760699.085669:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 20307968<>20312064 00000080:00200000:3.0:1789760699.137204:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4958, 4958] 00000080:00200000:3.0:1789760699.137221:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.137226:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.137241:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20312064 i_blocks 40967 00000020:00200000:3.0:1789760699.137255:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.137265:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.137298:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20312064, count: 4096 00000080:00200000:3.0:1789760699.137304:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20312064, bytes: 4096 00000080:00200000:3.0:1789760699.137314:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.137322:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.137324:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20312064, 20316160) 0 0 00000080:00200000:3.0:1789760699.137329:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.137345:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4959, 4959] 00020000:00200000:3.0:1789760699.137353:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20312064 chunk: [20312064, 20316160) 20316160, 4096 00020000:00200000:3.0:1789760699.137357:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.137375:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4959, 4959] 00000008:00010000:3.0:1789760699.137392:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.137397:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.137409:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20312064, 20316160) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.137423:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20312064=0x135f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.137478:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.137482:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20316160 00000020:00010000:3.0:1789760699.137490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.137494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.137497:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.137501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.137503:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.137510:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.137517:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.137533:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760699.137655:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 20312064<>20316160 00000080:00200000:3.0:1789760699.171399:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4959, 4959] 00000080:00200000:3.0:1789760699.171416:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.171422:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.171438:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20316160 i_blocks 40967 00000020:00200000:3.0:1789760699.171441:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.171451:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.171483:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20316160, count: 4096 00000080:00200000:3.0:1789760699.171491:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20316160, bytes: 4096 00000080:00200000:3.0:1789760699.171500:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.171509:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.171512:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20316160, 20320256) 0 0 00000080:00200000:3.0:1789760699.171517:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.171534:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4960, 4960] 00020000:00200000:3.0:1789760699.171543:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20316160 chunk: [20316160, 20320256) 20320256, 4096 00020000:00200000:3.0:1789760699.171549:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.171571:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4960, 4960] 00000008:00010000:3.0:1789760699.171590:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.171595:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.171606:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20316160, 20320256) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.171620:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20316160=0x1360000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.171680:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.171682:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20320256 00000020:00010000:3.0:1789760699.171691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.171694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.171697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.171701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.171704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.171712:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.171719:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.171736:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760699.171863:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 20316160<>20320256 00000080:00200000:3.0:1789760699.221601:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4960, 4960] 00000080:00200000:3.0:1789760699.221616:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.221622:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.221638:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20320256 i_blocks 40967 00000020:00200000:3.0:1789760699.221641:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.221652:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.221687:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20320256, count: 4096 00000080:00200000:3.0:1789760699.221695:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20320256, bytes: 4096 00000080:00200000:3.0:1789760699.221707:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.221717:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.221719:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20320256, 20324352) 0 0 00000080:00200000:3.0:1789760699.221724:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.221741:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4961, 4961] 00020000:00200000:3.0:1789760699.221749:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20320256 chunk: [20320256, 20324352) 20324352, 4096 00020000:00200000:3.0:1789760699.221753:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.221772:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4961, 4961] 00000008:00010000:3.0:1789760699.221789:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.221794:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.221805:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20320256, 20324352) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.221818:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20320256=0x1361000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.221881:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.221884:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20324352 00000020:00010000:3.0:1789760699.221892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.221895:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.221898:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.221975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.221978:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.221986:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.221993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.222009:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760699.222130:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 20320256<>20324352 00000080:00200000:3.0:1789760699.281445:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4961, 4961] 00000080:00200000:3.0:1789760699.281461:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.281468:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.281484:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20324352 i_blocks 40967 00000020:00200000:3.0:1789760699.281488:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.281497:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.281528:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20324352, count: 4096 00000080:00200000:3.0:1789760699.281536:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20324352, bytes: 4096 00000080:00200000:3.0:1789760699.281547:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.281556:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.281558:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20324352, 20328448) 0 0 00000080:00200000:3.0:1789760699.281564:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.281589:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4962, 4962] 00020000:00200000:3.0:1789760699.281598:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20324352 chunk: [20324352, 20328448) 20328448, 4096 00020000:00200000:3.0:1789760699.281602:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.281622:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4962, 4962] 00000008:00010000:3.0:1789760699.281640:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.281646:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.281658:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20324352, 20328448) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.281674:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20324352=0x1362000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.281730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.281733:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20328448 00000020:00010000:3.0:1789760699.281742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.281745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.281749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.281753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.281757:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.281764:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.281771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.281787:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760699.281913:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 20324352<>20328448 00000080:00200000:3.0:1789760699.412949:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4962, 4962] 00000080:00200000:3.0:1789760699.412966:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.412973:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.412996:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20328448 i_blocks 40967 00000020:00200000:3.0:1789760699.413000:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.413009:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.413041:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20328448, count: 4096 00000080:00200000:3.0:1789760699.413047:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20328448, bytes: 4096 00000080:00200000:3.0:1789760699.413057:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.413065:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.413067:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20328448, 20332544) 0 0 00000080:00200000:3.0:1789760699.413072:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.413087:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4963, 4963] 00020000:00200000:3.0:1789760699.413095:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20328448 chunk: [20328448, 20332544) 20332544, 4096 00020000:00200000:3.0:1789760699.413100:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.413119:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4963, 4963] 00000008:00010000:3.0:1789760699.413135:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.413140:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.413150:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20328448, 20332544) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.413197:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20328448=0x1363000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.413249:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.413252:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20332544 00000020:00010000:3.0:1789760699.413261:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.413264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.413267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.413271:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.413274:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.413282:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.413289:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.413307:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760699.413429:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 20328448<>20332544 00000080:00200000:3.0:1789760699.523370:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4963, 4963] 00000080:00200000:3.0:1789760699.523386:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.523393:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.523409:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20332544 i_blocks 40967 00000020:00200000:3.0:1789760699.523412:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.523422:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.523453:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20332544, count: 4096 00000080:00200000:3.0:1789760699.523460:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20332544, bytes: 4096 00000080:00200000:3.0:1789760699.523470:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.523480:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.523482:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20332544, 20336640) 0 0 00000080:00200000:3.0:1789760699.523487:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.523503:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4964, 4964] 00020000:00200000:3.0:1789760699.523511:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20332544 chunk: [20332544, 20336640) 20336640, 4096 00020000:00200000:3.0:1789760699.523516:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.523535:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4964, 4964] 00000008:00010000:3.0:1789760699.523551:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.523556:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.523567:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20332544, 20336640) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.523580:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20332544=0x1364000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.523636:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.523639:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20336640 00000020:00010000:3.0:1789760699.523648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.523651:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.523653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.523657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.523660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.523667:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.523674:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.523690:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760699.523809:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 20332544<>20336640 00000100:00100000:3.0:1789760699.523852:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760699.523868:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495712640:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760699.571655:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:3.0:1789760699.571678:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760699.571692:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760699.572074:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:3.0:1789760699.572286:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495712640:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760699.572803:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495712768:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760699.667763:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4967, 4967] 00000080:00200000:3.0:1789760699.667783:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.667790:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.667808:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20348928 i_blocks 40967 00000020:00200000:3.0:1789760699.667813:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.667825:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.668023:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20348928, count: 4096 00000080:00200000:3.0:1789760699.668033:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20348928, bytes: 4096 00000080:00200000:3.0:1789760699.668148:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.668158:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.668161:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20348928, 20353024) 0 0 00000080:00200000:3.0:1789760699.668165:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.668199:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4968, 4968] 00020000:00200000:3.0:1789760699.668208:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20348928 chunk: [20348928, 20353024) 20353024, 4096 00020000:00200000:3.0:1789760699.668213:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.668236:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4968, 4968] 00000008:00010000:3.0:1789760699.669327:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.669335:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.669349:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20348928, 20353024) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.669366:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20348928=0x1368000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.669509:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.669512:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20353024 00000020:00010000:3.0:1789760699.669521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.669525:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.669528:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.669532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.669535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.669545:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.669554:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.669584:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760699.669827:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 20348928<>20353024 00000080:00200000:3.0:1789760699.722957:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4968, 4968] 00000080:00200000:3.0:1789760699.722984:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.722991:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.723009:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20353024 i_blocks 40967 00000020:00200000:3.0:1789760699.723015:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.723032:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.723065:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20353024, count: 4096 00000080:00200000:3.0:1789760699.723074:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20353024, bytes: 4096 00000080:00200000:3.0:1789760699.723089:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.723099:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.723101:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20353024, 20357120) 0 0 00000080:00200000:3.0:1789760699.723105:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.723120:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4969, 4969] 00020000:00200000:3.0:1789760699.723131:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20353024 chunk: [20353024, 20357120) 20357120, 4096 00020000:00200000:3.0:1789760699.723135:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.849040:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4970, 4970] 00000080:00200000:3.0:1789760699.849058:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.849064:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.849079:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20361216 i_blocks 40967 00000020:00200000:3.0:1789760699.849083:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.849093:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.849127:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20361216, count: 4096 00000080:00200000:3.0:1789760699.849135:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20361216, bytes: 4096 00000080:00200000:3.0:1789760699.849145:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.849153:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.849155:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20361216, 20365312) 0 0 00000080:00200000:3.0:1789760699.849159:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.849224:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4971, 4971] 00020000:00200000:3.0:1789760699.849233:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20361216 chunk: [20361216, 20365312) 20365312, 4096 00020000:00200000:3.0:1789760699.849245:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.849266:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4971, 4971] 00000008:00010000:3.0:1789760699.849284:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.849290:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.849301:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20361216, 20365312) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.849315:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20361216=0x136b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.849372:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.849375:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20365312 00000020:00010000:3.0:1789760699.849384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.849387:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.849390:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.849394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.849397:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.849404:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.849410:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.849427:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760699.849561:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 20361216<>20365312 00000100:00100000:3.0:1789760699.850163:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760699.850302:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495713536:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760699.953698:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760699.953723:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495713792:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760699.983659:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4973, 4973] 00000080:00200000:3.0:1789760699.983676:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.983683:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.983698:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20373504 i_blocks 40967 00000020:00200000:3.0:1789760699.983702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.983711:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.983747:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20373504, count: 4096 00000080:00200000:3.0:1789760699.983755:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20373504, bytes: 4096 00000080:00200000:3.0:1789760699.983765:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.983774:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.983776:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20373504, 20377600) 0 0 00000080:00200000:3.0:1789760699.983781:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.983797:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4974, 4974] 00020000:00200000:3.0:1789760699.983805:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20373504 chunk: [20373504, 20377600) 20377600, 4096 00020000:00200000:3.0:1789760699.983810:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.983828:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4974, 4974] 00000008:00010000:3.0:1789760699.983845:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.983850:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760699.983988:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20373504, 20377600) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.984002:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20373504=0x136e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.984053:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.984056:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20377600 00000020:00010000:3.0:1789760699.984064:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.984069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.984072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.984075:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.984079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.984087:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.984093:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.984109:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760699.984306:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 20373504<>20377600 00000080:00200000:3.0:1789760700.254798:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4980, 4980] 00000080:00200000:3.0:1789760700.254828:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.254835:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.254858:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20402176 i_blocks 40967 00000020:00200000:3.0:1789760700.254862:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.254872:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.254906:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20402176, count: 4096 00000080:00200000:3.0:1789760700.254913:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20402176, bytes: 4096 00000080:00200000:3.0:1789760700.254932:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.254940:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.254943:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20402176, 20406272) 0 0 00000080:00200000:3.0:1789760700.254947:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.254964:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4981, 4981] 00020000:00200000:3.0:1789760700.254972:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20402176 chunk: [20402176, 20406272) 20406272, 4096 00020000:00200000:3.0:1789760700.254977:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.254997:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4981, 4981] 00000008:00010000:3.0:1789760700.255014:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.255020:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760700.255033:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20402176, 20406272) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.255048:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20402176=0x1375000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.255108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.255111:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20406272 00000020:00010000:3.0:1789760700.255124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.255128:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.255131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.255134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.255137:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.255145:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.255152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.255216:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760700.255349:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 20402176<>20406272 00000080:00200000:3.0:1789760700.350825:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4981, 4981] 00000080:00200000:3.0:1789760700.350841:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.350848:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.350864:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20406272 i_blocks 40967 00000020:00200000:3.0:1789760700.350868:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.350877:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.350910:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20406272, count: 4096 00000080:00200000:3.0:1789760700.350928:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20406272, bytes: 4096 00000080:00200000:3.0:1789760700.350939:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.350948:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.350950:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20406272, 20410368) 0 0 00000080:00200000:3.0:1789760700.350955:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.350973:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4982, 4982] 00020000:00200000:3.0:1789760700.350981:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20406272 chunk: [20406272, 20410368) 20410368, 4096 00020000:00200000:3.0:1789760700.350986:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.351004:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4982, 4982] 00000008:00010000:3.0:1789760700.351021:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.351027:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760700.351039:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20406272, 20410368) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.351054:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20406272=0x1376000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.351114:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.351116:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20410368 00000020:00010000:3.0:1789760700.351124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.351128:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.351131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.351134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.351138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.351145:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.351152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.351326:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760700.351449:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 20406272<>20410368 00000100:00100000:3.0:1789760700.491432:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760700.492493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495715200:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760700.545222:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972301 gen 2 00000008:00010000:3.0:1789760700.545273:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ecc0 00000020:00200000:3.0:1789760700.545281:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.545292:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.545315:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ecc0 00000100:00100000:3.0:1789760700.545344:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495715200:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760700.618514:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4986, 4986] 00000080:00200000:3.0:1789760700.618531:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.618537:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.618553:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20426752 i_blocks 40967 00000020:00200000:3.0:1789760700.618557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.618567:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.618599:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20426752, count: 4096 00000080:00200000:3.0:1789760700.618606:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20426752, bytes: 4096 00000080:00200000:3.0:1789760700.618616:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.618626:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.618697:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20426752, 20430848) 0 0 00000080:00200000:3.0:1789760700.618702:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.618720:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4987, 4987] 00020000:00200000:3.0:1789760700.618729:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20426752 chunk: [20426752, 20430848) 20430848, 4096 00020000:00200000:3.0:1789760700.618734:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.618754:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4987, 4987] 00000008:00010000:3.0:1789760700.618772:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.618888:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760700.618901:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20426752, 20430848) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.618925:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20426752=0x137b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.618984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.618987:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20430848 00000020:00010000:3.0:1789760700.618996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.619000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.619003:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.619006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.619009:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.619017:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.619023:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.619042:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760700.619191:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 20426752<>20430848 00000080:00200000:3.0:1789760700.687824:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4987, 4987] 00000080:00200000:3.0:1789760700.687851:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.687857:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.687873:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20430848 i_blocks 40967 00000020:00200000:3.0:1789760700.687877:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.687887:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.687938:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20430848, count: 4096 00000080:00200000:3.0:1789760700.687947:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20430848, bytes: 4096 00000080:00200000:3.0:1789760700.687956:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.687965:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.687968:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20430848, 20434944) 0 0 00000080:00200000:3.0:1789760700.687972:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.687989:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4988, 4988] 00020000:00200000:3.0:1789760700.687996:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20430848 chunk: [20430848, 20434944) 20434944, 4096 00020000:00200000:3.0:1789760700.688000:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.688019:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4988, 4988] 00000008:00010000:3.0:1789760700.688038:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.688044:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760700.688055:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20430848, 20434944) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.688447:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20430848=0x137c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.688518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.688521:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20434944 00000020:00010000:3.0:1789760700.688530:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.688533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.688537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.688540:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.688543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.688551:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.688557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.688576:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760700.688701:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 20430848<>20434944 00000100:00100000:3.0:1789760700.688750:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760700.688768:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495715712:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760700.736081:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:3.0:1789760700.736100:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.736113:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.736142:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:3.0:1789760700.736211:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495715712:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760700.914426:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [4991, 4991] 00000080:00200000:3.0:1789760700.914455:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.914468:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.914485:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20447232 i_blocks 40967 00000020:00200000:3.0:1789760700.914489:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.914499:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.914535:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20447232, count: 4096 00000080:00200000:3.0:1789760700.914543:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20447232, bytes: 4096 00000080:00200000:3.0:1789760700.914557:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.914567:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.914569:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20447232, 20451328) 0 0 00000080:00200000:3.0:1789760700.914575:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.914592:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [4992, 4992] 00020000:00200000:3.0:1789760700.914600:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20447232 chunk: [20447232, 20451328) 20451328, 4096 00020000:00200000:3.0:1789760700.914605:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.914627:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [4992, 4992] 00000008:00010000:3.0:1789760700.914645:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.914650:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760700.914663:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20447232, 20451328) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.914677:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20447232=0x1380000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.914740:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.914744:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20451328 00000020:00010000:3.0:1789760700.914752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.914755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.914758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.914762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.914765:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.914772:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.914779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.914797:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760700.914958:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 20447232<>20451328 00000100:00100000:3.0:1789760700.915012:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495716224:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760700.975256:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972309 gen 2 00000008:00010000:3.0:1789760700.975312:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:3.0:1789760700.975326:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.975337:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.975364:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:3.0:1789760700.975402:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495716224:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760701.739383:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5001, 5001] 00000080:00200000:3.0:1789760701.739406:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.739413:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.739428:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20488192 i_blocks 40967 00000020:00200000:3.0:1789760701.739432:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.739442:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.739479:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20488192, count: 4096 00000080:00200000:3.0:1789760701.739486:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20488192, bytes: 4096 00000080:00200000:3.0:1789760701.739498:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.739508:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.739511:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20488192, 20492288) 0 0 00000080:00200000:3.0:1789760701.739515:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.739534:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5002, 5002] 00020000:00200000:3.0:1789760701.739543:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20488192 chunk: [20488192, 20492288) 20492288, 4096 00020000:00200000:3.0:1789760701.739548:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.739570:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5002, 5002] 00000008:00010000:3.0:1789760701.739589:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.739594:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760701.739604:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20488192, 20492288) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.739618:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20488192=0x138a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.739698:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.739702:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20492288 00000020:00010000:3.0:1789760701.739711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.739714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.739718:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.739722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.739725:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.739733:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.739741:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.739762:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760701.740150:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 20488192<>20492288 00000100:00100000:3.0:1789760701.740371:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760701.740394:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495717504:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760701.877255:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972319 gen 2 00000008:00010000:3.0:1789760701.877323:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:3.0:1789760701.877332:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.877343:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.877363:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:3.0:1789760701.877394:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495717504:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760702.082724:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495717888:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760702.197154:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972322 gen 2 00000008:00010000:3.0:1789760702.197252:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760702.197260:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760702.197273:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760702.197296:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760702.197325:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495717888:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760702.198123:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495718016:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760702.257515:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972323 gen 2 00000008:00010000:3.0:1789760702.257571:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760702.257581:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760702.257593:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760702.257618:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760702.257651:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495718016:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760702.698030:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495719168:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760702.766732:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972329 gen 2 00000008:00010000:3.0:1789760702.766777:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760702.766785:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760702.766796:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760702.766819:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760702.766848:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495719168:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760702.767930:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760702.767952:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495719296:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760702.858304:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972330 gen 2 00000008:00010000:3.0:1789760702.858355:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760702.858365:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760702.858377:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760702.858401:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760702.858431:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495719296:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760703.254154:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495719936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760703.364733:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972335 gen 2 00000008:00010000:3.0:1789760703.364799:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760703.364808:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760703.364819:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760703.364841:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760703.364870:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495719936:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760703.371480:0:468:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1789760703.371503:0:468:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760703.371538:0:468:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760704.387125:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760704.387152:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495722240:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760704.419164:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972353 gen 2 00000008:00010000:3.0:1789760704.419275:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760704.419284:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760704.419296:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760704.419321:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760704.419349:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495722240:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760704.420688:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495722368:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760704.826045:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760704.826070:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495723648:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760704.919867:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972364 gen 2 00000008:00010000:3.0:1789760704.919935:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760704.919946:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760704.919958:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760704.919995:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760704.920031:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495723648:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760704.921701:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495723776:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760705.000866:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972365 gen 2 00000008:00010000:3.0:1789760705.000924:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760705.000933:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.000944:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.000969:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760705.001003:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495723776:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760705.498397:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5059, 5059] 00000080:00200000:3.0:1789760705.498416:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760705.498422:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760705.498437:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20725760 i_blocks 40967 00000020:00200000:3.0:1789760705.498441:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760705.498451:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760705.498486:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20725760, count: 4096 00000080:00200000:3.0:1789760705.498493:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20725760, bytes: 4096 00000080:00200000:3.0:1789760705.498504:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760705.498512:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760705.498515:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20725760, 20729856) 0 0 00000080:00200000:3.0:1789760705.498519:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760705.498537:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5060, 5060] 00020000:00200000:3.0:1789760705.498547:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20725760 chunk: [20725760, 20729856) 20729856, 4096 00020000:00200000:3.0:1789760705.498551:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760705.498570:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5060, 5060] 00000008:00010000:3.0:1789760705.498588:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760705.498593:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760705.498603:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20725760, 20729856) append 0 append lockpos 0 00000080:00200000:3.0:1789760705.498617:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20725760=0x13c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760705.498676:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760705.498679:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20729856 00000020:00010000:3.0:1789760705.498688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760705.498691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760705.498694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760705.498698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760705.498701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760705.498709:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760705.498716:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760705.498732:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760705.498854:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 20725760<>20729856 00000080:00200000:3.0:1789760705.534531:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5060, 5060] 00000080:00200000:3.0:1789760705.534547:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760705.534553:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760705.534568:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20729856 i_blocks 40967 00000020:00200000:3.0:1789760705.534571:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760705.534581:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760705.534612:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20729856, count: 4096 00000080:00200000:3.0:1789760705.534619:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20729856, bytes: 4096 00000080:00200000:3.0:1789760705.534628:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760705.534637:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760705.534639:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20729856, 20733952) 0 0 00000080:00200000:3.0:1789760705.534643:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760705.534663:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5061, 5061] 00020000:00200000:3.0:1789760705.534672:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20729856 chunk: [20729856, 20733952) 20733952, 4096 00020000:00200000:3.0:1789760705.534677:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760705.534697:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5061, 5061] 00000008:00010000:3.0:1789760705.534715:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760705.534721:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760705.534733:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20729856, 20733952) append 0 append lockpos 0 00000080:00200000:3.0:1789760705.534748:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20729856=0x13c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760705.534799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760705.534802:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20733952 00000020:00010000:3.0:1789760705.534810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760705.534813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760705.534816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760705.534820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760705.534822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760705.534829:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760705.534835:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760705.534921:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760705.535037:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 20729856<>20733952 00000100:00100000:3.0:1789760705.561011:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972378 gen 2 00000008:00010000:3.0:1789760705.561069:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760705.561076:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.561085:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.561106:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760705.561133:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495725440:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760705.622069:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495725824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760705.652399:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972381 gen 2 00000008:00010000:3.0:1789760705.652446:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760705.652454:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.652465:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.652488:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760705.652514:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495725824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760705.653594:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760705.653609:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495725952:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760705.683355:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760705.683369:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.683381:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.683449:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760705.683477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495725952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760705.684474:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760705.684491:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495726080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760705.711367:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972382 gen 2 00000100:00100000:3.0:1789760705.711406:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265500 x1876699495726080/t4294972383(4294972383) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760721 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760705.711453:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760705.711460:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.711471:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.711495:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760705.711518:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495726080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760705.712578:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495726208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760705.770464:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972384 gen 2 00000008:00010000:3.0:1789760705.770521:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760705.770529:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.770539:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.770564:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760705.770593:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495726208:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760705.810431:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5068, 5068] 00000080:00200000:3.0:1789760705.810448:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760705.810455:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760705.810472:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20762624 i_blocks 40967 00000020:00200000:3.0:1789760705.810476:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760705.810487:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760705.810522:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20762624, count: 4096 00000080:00200000:3.0:1789760705.810529:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20762624, bytes: 4096 00000080:00200000:3.0:1789760705.810541:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760705.810551:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760705.810553:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20762624, 20766720) 0 0 00000080:00200000:3.0:1789760705.810557:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760705.810573:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5069, 5069] 00020000:00200000:3.0:1789760705.810582:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20762624 chunk: [20762624, 20766720) 20766720, 4096 00020000:00200000:3.0:1789760705.810586:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760705.810606:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5069, 5069] 00000008:00010000:3.0:1789760705.810625:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760705.810642:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760705.810655:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20762624, 20766720) append 0 append lockpos 0 00000080:00200000:3.0:1789760705.810671:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20762624=0x13cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760705.810728:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760705.810731:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20766720 00000020:00010000:3.0:1789760705.810740:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760705.810744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760705.810747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760705.810750:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760705.810753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760705.810761:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760705.810768:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760705.810785:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760705.810931:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 20762624<>20766720 00000100:00100000:3.0:1789760705.810988:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760705.811007:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495726464:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760705.858287:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760705.858301:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.858312:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.858337:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760705.858367:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495726464:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760706.044661:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5072, 5072] 00000080:00200000:3.0:1789760706.044677:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760706.044683:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760706.044699:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20779008 i_blocks 40967 00000020:00200000:3.0:1789760706.044702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760706.044711:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760706.044745:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20779008, count: 4096 00000080:00200000:3.0:1789760706.044752:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20779008, bytes: 4096 00000080:00200000:3.0:1789760706.044762:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760706.044771:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760706.044773:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20779008, 20783104) 0 0 00000080:00200000:3.0:1789760706.044777:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760706.044793:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5073, 5073] 00020000:00200000:3.0:1789760706.044801:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20779008 chunk: [20779008, 20783104) 20783104, 4096 00020000:00200000:3.0:1789760706.044806:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760706.044824:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5073, 5073] 00000008:00010000:3.0:1789760706.044840:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760706.044845:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760706.044870:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20779008, 20783104) append 0 append lockpos 0 00000080:00200000:3.0:1789760706.044895:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20779008=0x13d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760706.044949:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760706.044952:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20783104 00000020:00010000:3.0:1789760706.044960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760706.044963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760706.044966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760706.044969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760706.044972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760706.044979:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760706.044985:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760706.045001:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760706.045117:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 20779008<>20783104 00000100:00100000:3.0:1789760706.045161:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760706.045224:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495726976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760706.077681:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972390 gen 2 00000008:00010000:3.0:1789760706.077742:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760706.077751:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.077764:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.077867:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760706.077908:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495726976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760706.079739:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495727104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760706.117822:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972391 gen 2 00000008:00010000:3.0:1789760706.117890:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760706.117899:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.117909:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.117932:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760706.117959:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495727104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760706.118868:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495727232:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760706.172082:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972392 gen 2 00000008:00010000:3.0:1789760706.172145:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760706.172154:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.172166:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.172229:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760706.172259:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495727232:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760706.175285:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760706.175316:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495727360:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760706.215896:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760706.215911:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.215923:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.215949:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760706.215979:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495727360:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760706.216983:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495727488:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760706.247769:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5077, 5077] 00000080:00200000:3.0:1789760706.247787:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760706.247794:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760706.248693:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495727616:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760706.284962:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972395 gen 2 00000008:00010000:3.0:1789760706.285011:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760706.285019:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.285031:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.285053:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760706.285079:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495727616:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.077912:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495729664:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760707.109997:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760707.110012:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.110022:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.110044:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760707.110070:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495729664:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.110858:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760707.110882:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495729792:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.140219:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972411 gen 2 00000100:00100000:3.0:1789760707.140256:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264e00 x1876699495729792/t4294972412(4294972412) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760723 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760707.140303:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760707.140310:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.140320:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.140342:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760707.140364:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495729792:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.141015:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495729920:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.173017:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972413 gen 2 00000008:00010000:3.0:1789760707.173081:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760707.173089:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.173099:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.173121:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760707.173153:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495729920:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.174025:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495730048:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.206521:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972414 gen 2 00000008:00010000:3.0:1789760707.206565:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760707.206573:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.206583:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.206605:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760707.206631:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495730048:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.207696:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730176:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760707.237066:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760707.237081:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.237092:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.237115:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760707.237144:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730176:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.238443:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760707.238464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730304:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.267341:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972416 gen 2 00000008:00010000:3.0:1789760707.267399:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760707.267408:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.267418:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.267440:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760707.267468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730304:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.268196:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730432:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.293925:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972417 gen 2 00000008:00010000:3.0:1789760707.293973:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760707.293981:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.293991:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.294014:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760707.294042:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730432:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.294802:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760707.294820:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.329497:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972418 gen 2 00000008:00010000:3.0:1789760707.329542:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760707.329550:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.329560:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.329580:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760707.329607:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.330651:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.330672:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495730688:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760707.359591:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760707.359605:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.359617:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.359635:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760707.359664:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495730688:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.360831:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760707.360858:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.399328:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972420 gen 2 00000008:00010000:3.0:1789760707.399388:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760707.399398:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.399408:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.399432:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760707.399460:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495730816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.400894:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.400919:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495730944:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.443061:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972421 gen 2 00000008:00010000:3.0:1789760707.443109:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760707.443117:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.443128:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.443151:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760707.443222:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495730944:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.444511:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760707.444532:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495731072:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.496797:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972422 gen 2 00000008:00010000:3.0:1789760707.496859:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760707.496877:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.496888:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.496911:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760707.496946:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495731072:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.498387:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.498412:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495731584:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.616936:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972423 gen 2 00000008:00010000:3.0:1789760707.616980:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760707.616988:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.616999:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.617023:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760707.617050:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495731584:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.617779:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760707.617797:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495731712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.662485:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972424 gen 2 00000008:00010000:3.0:1789760707.662532:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760707.662540:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.662550:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.662572:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760707.662598:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495731712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.664013:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.664035:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495731840:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.712544:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972425 gen 2 00000008:00010000:3.0:1789760707.712589:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760707.712598:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.712608:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.712630:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760707.712658:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495731840:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.713555:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495731968:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760707.770345:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972426 gen 2 00000008:00010000:3.0:1789760707.770394:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760707.770404:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.770417:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.770441:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760707.770472:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495731968:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760707.881531:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5110, 5110] 00000080:00200000:3.0:1789760707.881557:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760707.881564:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760707.970977:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5111, 5111] 00000080:00200000:3.0:1789760707.971038:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760707.971045:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760707.972017:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495732352:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760708.154346:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5114, 5114] 00000080:00200000:3.0:1789760708.154524:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760708.154532:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760708.154550:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20951040 i_blocks 40967 00000020:00200000:3.0:1789760708.154555:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760708.154565:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760708.154687:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20951040, count: 4096 00000080:00200000:3.0:1789760708.154697:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20951040, bytes: 4096 00000080:00200000:3.0:1789760708.154711:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760708.154720:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760708.154722:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20951040, 20955136) 0 0 00000080:00200000:3.0:1789760708.154727:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760708.154809:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5115, 5115] 00020000:00200000:3.0:1789760708.154818:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20951040 chunk: [20951040, 20955136) 20955136, 4096 00020000:00200000:3.0:1789760708.154823:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760708.154846:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5115, 5115] 00000008:00010000:3.0:1789760708.154877:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760708.154883:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760708.154895:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20951040, 20955136) append 0 append lockpos 0 00000080:00200000:3.0:1789760708.154911:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=20951040=0x13fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760708.154981:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760708.154984:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20955136 00000020:00010000:3.0:1789760708.154994:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760708.154997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760708.155000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760708.155004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760708.155007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760708.155014:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760708.155021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760708.155051:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760708.155805:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 20951040<>20955136 00000100:00100000:3.0:1789760708.155872:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760708.155897:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495732736:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760708.240378:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760708.240394:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.240406:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.240431:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760708.240460:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495732736:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760708.278621:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495732992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760708.317727:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972434 gen 2 00000008:00010000:3.0:1789760708.317774:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760708.317783:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.317794:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.317818:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760708.317868:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495732992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760709.955483:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760709.955506:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495736320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760710.038012:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972460 gen 2 00000008:00010000:3.0:1789760710.038063:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760710.038072:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.038084:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.038106:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760710.038145:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495736320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760710.039821:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495736448:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760710.094861:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760710.094877:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.094889:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.094913:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760710.094941:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495736448:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760710.095838:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760710.095865:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495736576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760710.144569:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972462 gen 2 00000008:00010000:3.0:1789760710.144636:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760710.144644:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.144656:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.144682:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760710.144711:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495736576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760710.145559:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495736704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760710.215241:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972463 gen 2 00000008:00010000:3.0:1789760710.215292:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760710.215300:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.215312:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.215324:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760710.215349:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495736704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.345891:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495739008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.432486:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972481 gen 2 00000008:00010000:3.0:1789760711.432543:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760711.432552:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.432563:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.432589:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760711.432616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495739008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.434092:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760711.434117:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495739136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.478644:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972482 gen 2 00000008:00010000:3.0:1789760711.478690:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760711.478697:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.478707:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.478727:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760711.478750:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495739136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.480678:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760711.480707:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495739264:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.508477:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972483 gen 2 00000008:00010000:3.0:1789760711.508525:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760711.508533:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.508544:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.508566:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760711.508594:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495739264:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.509424:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495739392:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.591840:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495739648:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.633119:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972486 gen 2 00000008:00010000:3.0:1789760711.633200:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760711.633210:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.633223:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.633364:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760711.633393:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495739648:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760711.949130:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760711.949303:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495740416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760712.005017:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972492 gen 2 00000008:00010000:3.0:1789760712.005083:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760712.005092:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.005102:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.005128:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760712.005200:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495740416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760712.005826:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760712.005859:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495740544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760712.051671:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972493 gen 2 00000008:00010000:3.0:1789760712.051723:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760712.051732:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.051744:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.051771:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760712.051803:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495740544:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760712.637870:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5183, 5183] 00000080:00200000:3.0:1789760712.637890:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760712.637897:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760712.637913:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21233664 i_blocks 43015 00000020:00200000:3.0:1789760712.637917:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760712.637938:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760712.637972:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21233664, count: 4096 00000080:00200000:3.0:1789760712.637979:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21233664, bytes: 4096 00000080:00200000:3.0:1789760712.637990:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760712.637999:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760712.638001:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21233664, 21237760) 0 0 00000080:00200000:3.0:1789760712.638006:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760712.638022:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5184, 5184] 00020000:00200000:3.0:1789760712.638030:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21233664 chunk: [21233664, 21237760) 21237760, 4096 00020000:00200000:3.0:1789760712.638034:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760712.638053:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5184, 5184] 00000008:00010000:3.0:1789760712.638069:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760712.638074:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760712.638086:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21233664, 21237760) append 0 append lockpos 0 00000080:00200000:3.0:1789760712.638100:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21233664=0x1440000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760712.638156:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760712.638159:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21237760 00000020:00010000:3.0:1789760712.638205:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760712.638209:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760712.638220:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760712.638224:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760712.638227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760712.638235:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760712.638241:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760712.638257:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760712.638390:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 21233664<>21237760 00000100:00100000:3.0:1789760712.638427:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760712.638444:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495741952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760712.682107:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972501 gen 2 00000008:00010000:3.0:1789760712.682211:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760712.682221:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.682234:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.682259:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760712.682291:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495741952:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760712.730235:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5185, 5185] 00000080:00200000:3.0:1789760712.730254:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760712.730260:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760712.730275:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21241856 i_blocks 43015 00000020:00200000:3.0:1789760712.730278:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760712.730287:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760712.730320:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21241856, count: 4096 00000080:00200000:3.0:1789760712.730328:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21241856, bytes: 4096 00000080:00200000:3.0:1789760712.730338:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760712.730346:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760712.730349:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21241856, 21245952) 0 0 00000080:00200000:3.0:1789760712.730353:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760712.730368:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5186, 5186] 00020000:00200000:3.0:1789760712.730377:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21241856 chunk: [21241856, 21245952) 21245952, 4096 00020000:00200000:3.0:1789760712.730381:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760712.730400:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5186, 5186] 00000008:00010000:3.0:1789760712.730417:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760712.730422:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760712.730433:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21241856, 21245952) append 0 append lockpos 0 00000080:00200000:3.0:1789760712.730446:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21241856=0x1442000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760712.730493:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760712.730496:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21245952 00000020:00010000:3.0:1789760712.730504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760712.730508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760712.730511:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760712.730515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760712.730517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760712.730525:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760712.730532:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760712.808792:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5186, 5186] 00000080:00200000:3.0:1789760712.808819:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760712.808833:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760712.808856:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21245952 i_blocks 43015 00000020:00200000:3.0:1789760712.808861:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760712.808870:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760712.808903:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21245952, count: 4096 00000080:00200000:3.0:1789760712.808910:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21245952, bytes: 4096 00000080:00200000:3.0:1789760712.808923:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760712.808931:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760712.808933:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21245952, 21250048) 0 0 00000080:00200000:3.0:1789760712.808938:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760712.808953:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5187, 5187] 00020000:00200000:3.0:1789760712.808960:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21245952 chunk: [21245952, 21250048) 21250048, 4096 00020000:00200000:3.0:1789760712.808965:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760712.808985:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5187, 5187] 00000008:00010000:3.0:1789760712.809002:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760712.809008:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760712.809020:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21245952, 21250048) append 0 append lockpos 0 00000080:00200000:3.0:1789760712.809034:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21245952=0x1443000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760712.809091:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760712.809094:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21250048 00000020:00010000:3.0:1789760712.809102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760712.809105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760712.809108:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760712.809112:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760712.809115:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760712.809123:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760712.809129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760712.809492:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760712.809613:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 21245952<>21250048 00000100:00100000:3.0:1789760713.136971:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495742976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.169656:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972509 gen 2 00000008:00010000:3.0:1789760713.169710:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760713.169718:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.169729:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.169756:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760713.169797:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495742976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.171114:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495743104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.216374:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972510 gen 2 00000008:00010000:3.0:1789760713.216422:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760713.216430:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.216442:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.216466:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760713.216490:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495743104:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760713.245620:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5194, 5194] 00000080:00200000:3.0:1789760713.245648:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760713.245656:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760713.245672:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21278720 i_blocks 43015 00000020:00200000:3.0:1789760713.245676:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760713.245685:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760713.245720:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21278720, count: 4096 00000080:00200000:3.0:1789760713.245727:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21278720, bytes: 4096 00000080:00200000:3.0:1789760713.245737:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760713.245746:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760713.245749:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21278720, 21282816) 0 0 00000080:00200000:3.0:1789760713.245754:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760713.245772:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5195, 5195] 00020000:00200000:3.0:1789760713.245780:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21278720 chunk: [21278720, 21282816) 21282816, 4096 00020000:00200000:3.0:1789760713.245784:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760713.245816:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5195, 5195] 00000008:00010000:3.0:1789760713.245843:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760713.245849:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760713.245861:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21278720, 21282816) append 0 append lockpos 0 00000080:00200000:3.0:1789760713.245875:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21278720=0x144b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760713.245934:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760713.245937:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21282816 00000020:00010000:3.0:1789760713.245946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760713.245949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.245953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.245956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760713.245959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760713.245966:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760713.245973:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760713.245988:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760713.246116:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 21278720<>21282816 00000080:00200000:3.0:1789760713.284346:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5195, 5195] 00000080:00200000:3.0:1789760713.284363:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760713.284369:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760713.284386:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21282816 i_blocks 43015 00000020:00200000:3.0:1789760713.284389:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760713.284399:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760713.284434:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21282816, count: 4096 00000080:00200000:3.0:1789760713.284441:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21282816, bytes: 4096 00000080:00200000:3.0:1789760713.284450:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760713.284461:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760713.284463:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21282816, 21286912) 0 0 00000080:00200000:3.0:1789760713.284467:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760713.284485:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5196, 5196] 00020000:00200000:3.0:1789760713.284493:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21282816 chunk: [21282816, 21286912) 21286912, 4096 00020000:00200000:3.0:1789760713.284498:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760713.284525:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5196, 5196] 00000008:00010000:3.0:1789760713.284542:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760713.284547:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760713.284558:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21282816, 21286912) append 0 append lockpos 0 00000080:00200000:3.0:1789760713.284572:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21282816=0x144c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760713.284626:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760713.284630:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21286912 00000020:00010000:3.0:1789760713.284638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760713.284642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.284645:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.284649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760713.284652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760713.284660:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760713.284667:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760713.284683:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760713.284805:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 21282816<>21286912 00000080:00200000:3.0:1789760713.324916:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5196, 5196] 00000080:00200000:3.0:1789760713.324933:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760713.324939:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760713.324956:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21286912 i_blocks 43015 00000020:00200000:3.0:1789760713.324959:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760713.324968:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760713.325001:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21286912, count: 4096 00000080:00200000:3.0:1789760713.325009:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21286912, bytes: 4096 00000080:00200000:3.0:1789760713.325020:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760713.325030:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760713.325032:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21286912, 21291008) 0 0 00000080:00200000:3.0:1789760713.325036:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760713.325053:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5197, 5197] 00020000:00200000:3.0:1789760713.325061:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21286912 chunk: [21286912, 21291008) 21291008, 4096 00020000:00200000:3.0:1789760713.325066:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760713.325085:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5197, 5197] 00000008:00010000:3.0:1789760713.325103:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760713.325110:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760713.325121:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21286912, 21291008) append 0 append lockpos 0 00000080:00200000:3.0:1789760713.325135:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21286912=0x144d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760713.325231:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760713.325234:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21291008 00000020:00010000:3.0:1789760713.325244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760713.325248:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.325251:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.325255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760713.325258:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760713.325267:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760713.325274:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760713.325290:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760713.325410:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 21286912<>21291008 00000100:00100000:3.0:1789760713.325454:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760713.325469:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495743616:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.358577:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972514 gen 2 00000008:00010000:3.0:1789760713.358629:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:3.0:1789760713.358639:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.358650:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.358679:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:3.0:1789760713.358711:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495743616:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760713.403580:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5198, 5198] 00000080:00200000:3.0:1789760713.403602:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760713.403608:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760713.403637:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21295104 i_blocks 43015 00000020:00200000:3.0:1789760713.403641:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760713.403651:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760713.403687:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21295104, count: 4096 00000080:00200000:3.0:1789760713.403696:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21295104, bytes: 4096 00000080:00200000:3.0:1789760713.403707:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760713.403717:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760713.403720:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21295104, 21299200) 0 0 00000080:00200000:3.0:1789760713.403725:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760713.403743:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5199, 5199] 00020000:00200000:3.0:1789760713.403752:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21295104 chunk: [21295104, 21299200) 21299200, 4096 00020000:00200000:3.0:1789760713.403757:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760713.403778:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5199, 5199] 00000008:00010000:3.0:1789760713.403797:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760713.403804:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760713.403825:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21295104, 21299200) append 0 append lockpos 0 00000080:00200000:3.0:1789760713.403842:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21295104=0x144f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760713.403905:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760713.403908:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21299200 00000020:00010000:3.0:1789760713.403916:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760713.403919:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.403929:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.403933:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760713.403935:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760713.403943:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760713.403949:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760713.403966:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760713.404494:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 21295104<>21299200 00000100:00100000:3.0:1789760713.404539:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760713.404564:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495743872:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.471540:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972516 gen 2 00000008:00010000:3.0:1789760713.471601:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:3.0:1789760713.471610:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.471620:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.471700:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:3.0:1789760713.471728:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495743872:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.476000:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495744000:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760713.547431:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760713.547451:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.547465:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.547592:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760713.547626:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495744000:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.549375:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744128:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.603800:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972517 gen 2 00000100:00100000:3.0:1789760713.603863:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265880 x1876699495744128/t4294972518(4294972518) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760729 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.603912:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760713.603920:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.603931:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.603954:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760713.603979:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744128:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.604766:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760713.604782:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744256:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.746381:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972519 gen 2 00000008:00010000:3.0:1789760713.746446:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760713.746455:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.746466:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.746493:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760713.746527:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744256:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.747535:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744384:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760713.815787:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760713.815820:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.815833:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.815856:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760713.815885:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744384:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.816856:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495744512:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.853644:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972520 gen 2 00000100:00100000:3.0:1789760713.853685:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495744512/t4294972521(4294972521) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760729 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.853736:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760713.853744:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.853755:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.853884:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760713.853909:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495744512:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.854961:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744640:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.896015:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972521 gen 2 00000100:00100000:3.0:1789760713.896056:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264700 x1876699495744640/t4294972522(4294972522) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760729 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.896108:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760713.896115:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.896126:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.896157:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760713.896338:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744640:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.897653:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495744768:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.932789:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972523 gen 2 00000008:00010000:3.0:1789760713.932872:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760713.932880:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.932891:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.932968:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760713.932999:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495744768:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760713.934562:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495744896:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760713.971497:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5207, 5207] 00000080:00200000:3.0:1789760713.971516:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760713.971522:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760713.971538:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21331968 i_blocks 43015 00000020:00200000:3.0:1789760713.971541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760713.971550:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760713.971585:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21331968, count: 4096 00000080:00200000:3.0:1789760713.971592:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21331968, bytes: 4096 00000080:00200000:3.0:1789760713.971602:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760713.971611:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760713.971613:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21331968, 21336064) 0 0 00000080:00200000:3.0:1789760713.971618:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760713.971634:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5208, 5208] 00020000:00200000:3.0:1789760713.971644:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21331968 chunk: [21331968, 21336064) 21336064, 4096 00020000:00200000:3.0:1789760713.971648:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760713.971670:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5208, 5208] 00000008:00010000:3.0:1789760713.971688:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760713.971694:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760713.971705:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21331968, 21336064) append 0 append lockpos 0 00000080:00200000:3.0:1789760713.971719:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21331968=0x1458000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760713.971775:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760713.971778:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21336064 00000020:00010000:3.0:1789760713.971786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760713.971800:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.971804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760713.971807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760713.971810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760713.971827:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760713.971834:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760713.971850:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760713.971971:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 21331968<>21336064 00000080:00200000:3.0:1789760714.015805:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5208, 5208] 00000080:00200000:3.0:1789760714.015832:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.015839:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.015866:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21336064 i_blocks 43015 00000020:00200000:3.0:1789760714.015870:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.015881:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.015919:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21336064, count: 4096 00000080:00200000:3.0:1789760714.015928:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21336064, bytes: 4096 00000080:00200000:3.0:1789760714.015938:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.015949:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.015951:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21336064, 21340160) 0 0 00000080:00200000:3.0:1789760714.015956:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.015975:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5209, 5209] 00020000:00200000:3.0:1789760714.015984:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21336064 chunk: [21336064, 21340160) 21340160, 4096 00020000:00200000:3.0:1789760714.015989:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.016009:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5209, 5209] 00000008:00010000:3.0:1789760714.016131:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.016146:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760714.016161:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21336064, 21340160) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.016279:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21336064=0x1459000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.016339:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.016341:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21340160 00000020:00010000:3.0:1789760714.016349:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.016353:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.016356:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.016359:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.016362:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.016369:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.016376:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.016393:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760714.016518:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 21336064<>21340160 00000080:00200000:3.0:1789760714.137802:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5210, 5210] 00000080:00200000:3.0:1789760714.137828:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.137835:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.137852:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21344256 i_blocks 43015 00000020:00200000:3.0:1789760714.137856:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.137865:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.137903:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21344256, count: 4096 00000080:00200000:3.0:1789760714.137909:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21344256, bytes: 4096 00000080:00200000:3.0:1789760714.137922:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.137928:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.137930:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21344256, 21348352) 0 0 00000080:00200000:3.0:1789760714.137934:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.137949:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5211, 5211] 00020000:00200000:3.0:1789760714.137957:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21344256 chunk: [21344256, 21348352) 21348352, 4096 00020000:00200000:3.0:1789760714.137960:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.137977:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5211, 5211] 00000008:00010000:3.0:1789760714.137990:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.137996:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760714.138006:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21344256, 21348352) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.138019:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21344256=0x145b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.138082:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.138085:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21348352 00000020:00010000:3.0:1789760714.138104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.138108:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.138112:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.138115:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.138118:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.138126:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.138133:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.138157:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760714.138479:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 21344256<>21348352 00000100:00100000:3.0:1789760714.138519:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760714.138539:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495745408:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760714.713557:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495746560:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760714.742740:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760714.742758:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.742769:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.742805:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760714.742846:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495746560:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760714.890211:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5221, 5221] 00000080:00200000:3.0:1789760714.890227:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.890236:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.890252:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21389312 i_blocks 43015 00000020:00200000:3.0:1789760714.890256:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.890265:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.890294:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21389312, count: 4096 00000080:00200000:3.0:1789760714.890301:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21389312, bytes: 4096 00000080:00200000:3.0:1789760714.890314:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.890323:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.890326:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21389312, 21393408) 0 0 00000080:00200000:3.0:1789760714.890330:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.890346:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5222, 5222] 00020000:00200000:3.0:1789760714.890354:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21389312 chunk: [21389312, 21393408) 21393408, 4096 00020000:00200000:3.0:1789760714.890359:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.890377:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5222, 5222] 00000008:00010000:3.0:1789760714.890393:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.890398:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760714.890409:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21389312, 21393408) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.890443:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21389312=0x1466000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.890498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.890500:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21393408 00000020:00010000:3.0:1789760714.890508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.890511:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.890514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.890518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.890521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.890528:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.890534:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.890551:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760714.890675:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 21389312<>21393408 00000100:00100000:3.0:1789760714.890712:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760714.890729:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495746816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760714.986480:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972539 gen 2 00000008:00010000:3.0:1789760714.986529:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760714.986538:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.986549:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.986575:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760714.986606:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495746816:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760715.556704:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5230, 5230] 00000080:00200000:3.0:1789760715.556722:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760715.556728:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760715.556745:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21426176 i_blocks 43015 00000020:00200000:3.0:1789760715.556749:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760715.556759:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760715.556797:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21426176, count: 4096 00000080:00200000:3.0:1789760715.556814:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21426176, bytes: 4096 00000080:00200000:3.0:1789760715.556824:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760715.556833:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760715.556835:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21426176, 21430272) 0 0 00000080:00200000:3.0:1789760715.556840:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760715.556917:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5231, 5231] 00020000:00200000:3.0:1789760715.556926:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21426176 chunk: [21426176, 21430272) 21430272, 4096 00020000:00200000:3.0:1789760715.556931:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760715.556950:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5231, 5231] 00000008:00010000:3.0:1789760715.556966:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760715.556971:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760715.556982:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21426176, 21430272) append 0 append lockpos 0 00000080:00200000:3.0:1789760715.556994:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21426176=0x146f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760715.557045:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760715.557047:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21430272 00000020:00010000:3.0:1789760715.557055:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760715.557058:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760715.557062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760715.557066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760715.557069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760715.557076:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760715.557082:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760715.557098:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760715.557255:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 21426176<>21430272 00000080:00200000:3.0:1789760715.607219:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5231, 5231] 00000080:00200000:3.0:1789760715.607236:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760715.607243:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760715.607258:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21430272 i_blocks 43015 00000020:00200000:3.0:1789760715.607261:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760715.607270:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760715.607307:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21430272, count: 4096 00000080:00200000:3.0:1789760715.607314:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21430272, bytes: 4096 00000080:00200000:3.0:1789760715.607325:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760715.607334:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760715.607337:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21430272, 21434368) 0 0 00000080:00200000:3.0:1789760715.607341:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760715.607357:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5232, 5232] 00020000:00200000:3.0:1789760715.607365:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21430272 chunk: [21430272, 21434368) 21434368, 4096 00020000:00200000:3.0:1789760715.607370:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760715.607390:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5232, 5232] 00000008:00010000:3.0:1789760715.607406:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760715.607412:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760715.607423:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21430272, 21434368) append 0 append lockpos 0 00000080:00200000:3.0:1789760715.607436:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21430272=0x1470000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760715.607491:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760715.607494:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21434368 00000020:00010000:3.0:1789760715.607502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760715.607505:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760715.607509:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760715.607512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760715.607514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760715.607522:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760715.607529:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760715.607545:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760715.607669:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 21430272<>21434368 00000080:00200000:3.0:1789760715.668622:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5232, 5232] 00000080:00200000:3.0:1789760715.668639:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760715.668645:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760715.668662:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21434368 i_blocks 43015 00000020:00200000:3.0:1789760715.668666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760715.668676:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760715.668710:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21434368, count: 4096 00000080:00200000:3.0:1789760715.668717:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21434368, bytes: 4096 00000080:00200000:3.0:1789760715.668728:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760715.668737:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760715.668740:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21434368, 21438464) 0 0 00000080:00200000:3.0:1789760715.668745:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760715.668762:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5233, 5233] 00020000:00200000:3.0:1789760715.668771:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21434368 chunk: [21434368, 21438464) 21438464, 4096 00020000:00200000:3.0:1789760715.668776:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760715.668796:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5233, 5233] 00000008:00010000:3.0:1789760715.668828:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760715.668834:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760715.668857:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21434368, 21438464) append 0 append lockpos 0 00000080:00200000:3.0:1789760715.668873:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21434368=0x1471000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760715.668935:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760715.668938:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21438464 00000020:00010000:3.0:1789760715.668947:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760715.668950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760715.668954:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760715.668958:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760715.668961:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760715.668969:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760715.668976:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760715.668995:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760715.669322:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 21434368<>21438464 00000080:00200000:3.0:1789760715.740981:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5233, 5233] 00000080:00200000:3.0:1789760715.740997:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760715.741003:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760715.741018:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21438464 i_blocks 43015 00000020:00200000:3.0:1789760715.741021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760715.741029:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760715.741060:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21438464, count: 4096 00000080:00200000:3.0:1789760715.741067:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21438464, bytes: 4096 00000080:00200000:3.0:1789760715.741076:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760715.741084:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760715.741086:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21438464, 21442560) 0 0 00000080:00200000:3.0:1789760715.741090:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760715.741106:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5234, 5234] 00020000:00200000:3.0:1789760715.741114:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21438464 chunk: [21438464, 21442560) 21442560, 4096 00020000:00200000:3.0:1789760715.741119:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760715.741222:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5234, 5234] 00000008:00010000:3.0:1789760715.741238:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760715.741243:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760715.741254:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21438464, 21442560) append 0 append lockpos 0 00000080:00200000:3.0:1789760715.741268:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21438464=0x1472000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760715.741321:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760715.741323:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21442560 00000020:00010000:3.0:1789760715.741333:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760715.741336:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760715.741340:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760715.741343:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760715.741346:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760715.741353:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760715.741360:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760715.741375:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760715.741502:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 21438464<>21442560 00000080:00200000:3.0:1789760715.808398:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5234, 5234] 00000080:00200000:3.0:1789760715.808419:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760715.808425:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760715.900236:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760715.900256:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495748736:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760715.965377:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972554 gen 2 00000008:00010000:3.0:1789760715.965431:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760715.965439:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.965451:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.965537:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760715.965569:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495748736:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760715.967446:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495748864:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760716.057399:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972555 gen 2 00000008:00010000:3.0:1789760716.057437:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760716.057444:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.057453:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.057478:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760716.057506:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495748864:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760716.117761:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5239, 5239] 00000080:00200000:3.0:1789760716.117774:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.117780:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.117791:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21463040 i_blocks 43015 00000020:00200000:3.0:1789760716.117794:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.117813:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.117870:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21463040, count: 4096 00000080:00200000:3.0:1789760716.117876:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21463040, bytes: 4096 00000080:00200000:3.0:1789760716.117884:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.117891:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.117893:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21463040, 21467136) 0 0 00000080:00200000:3.0:1789760716.117896:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.117908:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5240, 5240] 00020000:00200000:3.0:1789760716.117915:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21463040 chunk: [21463040, 21467136) 21467136, 4096 00020000:00200000:3.0:1789760716.117919:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.117935:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5240, 5240] 00000008:00010000:3.0:1789760716.117949:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.117954:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.117963:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21463040, 21467136) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.117973:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21463040=0x1478000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.118016:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.118019:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21467136 00000020:00010000:3.0:1789760716.118026:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.118029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.118031:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.118034:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.118036:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.118042:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.118047:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.118060:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760716.118153:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 21463040<>21467136 00000100:00100000:3.0:1789760716.118219:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760716.118232:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495749120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760716.167829:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972556 gen 2 00000100:00100000:3.0:1789760716.167860:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203144e00 x1876699495749120/t4294972557(4294972557) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760732 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760716.167900:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:3.0:1789760716.167905:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.167914:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.167939:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:3.0:1789760716.167961:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495749120:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760716.269387:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5241, 5241] 00000080:00200000:3.0:1789760716.269400:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.269405:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.269428:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21471232 i_blocks 43015 00000020:00200000:3.0:1789760716.269431:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.269439:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.269463:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21471232, count: 4096 00000080:00200000:3.0:1789760716.269469:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21471232, bytes: 4096 00000080:00200000:3.0:1789760716.269476:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.269484:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.269486:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21471232, 21475328) 0 0 00000080:00200000:3.0:1789760716.269489:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.269502:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5242, 5242] 00020000:00200000:3.0:1789760716.269509:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21471232 chunk: [21471232, 21475328) 21475328, 4096 00020000:00200000:3.0:1789760716.269512:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.269526:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5242, 5242] 00000008:00010000:3.0:1789760716.269540:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.269545:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.269555:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21471232, 21475328) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.269681:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21471232=0x147a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.269789:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.269792:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21475328 00000020:00010000:3.0:1789760716.269812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.269816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.269818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.269822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.269824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.269832:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.269838:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.269861:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760716.269956:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 21471232<>21475328 00000080:00200000:3.0:1789760716.356543:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5242, 5242] 00000080:00200000:3.0:1789760716.356559:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.356566:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.356582:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21475328 i_blocks 43015 00000020:00200000:3.0:1789760716.356586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.356596:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.356632:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21475328, count: 4096 00000080:00200000:3.0:1789760716.356639:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21475328, bytes: 4096 00000080:00200000:3.0:1789760716.356655:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.356664:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.356666:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21475328, 21479424) 0 0 00000080:00200000:3.0:1789760716.356671:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.356689:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5243, 5243] 00020000:00200000:3.0:1789760716.356697:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21475328 chunk: [21475328, 21479424) 21479424, 4096 00020000:00200000:3.0:1789760716.356702:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.356724:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5243, 5243] 00000008:00010000:3.0:1789760716.356744:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.356750:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.356762:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21475328, 21479424) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.357218:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21475328=0x147b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.357282:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.357286:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21479424 00000020:00010000:3.0:1789760716.357294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.357297:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.357300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.357303:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.357306:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.357314:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.357321:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.357342:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760716.357465:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 21475328<>21479424 00000080:00200000:3.0:1789760716.458402:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5243, 5243] 00000080:00200000:3.0:1789760716.458418:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.458424:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.458439:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21479424 i_blocks 43015 00000020:00200000:3.0:1789760716.458443:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.458452:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.458484:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21479424, count: 4096 00000080:00200000:3.0:1789760716.458490:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21479424, bytes: 4096 00000080:00200000:3.0:1789760716.458499:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.458507:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.458509:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21479424, 21483520) 0 0 00000080:00200000:3.0:1789760716.458514:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.458531:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5244, 5244] 00020000:00200000:3.0:1789760716.458539:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21479424 chunk: [21479424, 21483520) 21483520, 4096 00020000:00200000:3.0:1789760716.458543:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.458563:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5244, 5244] 00000008:00010000:3.0:1789760716.458581:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.458587:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.458598:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21479424, 21483520) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.458612:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21479424=0x147c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.458667:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.458671:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21483520 00000020:00010000:3.0:1789760716.458679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.458682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.458685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.458688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.458691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.458699:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.458705:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.458721:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760716.458942:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 21479424<>21483520 00000100:00100000:3.0:1789760716.458988:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760716.459005:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495749632:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760716.501235:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972561 gen 2 00000008:00010000:3.0:1789760716.501289:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:3.0:1789760716.501297:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.501309:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.501324:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:3.0:1789760716.501354:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495749632:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760716.555675:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5245, 5245] 00000080:00200000:3.0:1789760716.555693:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.555700:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.555714:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21487616 i_blocks 43015 00000020:00200000:3.0:1789760716.555718:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.555727:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.555763:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21487616, count: 4096 00000080:00200000:3.0:1789760716.555786:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21487616, bytes: 4096 00000080:00200000:3.0:1789760716.555808:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.555817:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.555819:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21487616, 21491712) 0 0 00000080:00200000:3.0:1789760716.555825:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.555842:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5246, 5246] 00020000:00200000:3.0:1789760716.555859:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21487616 chunk: [21487616, 21491712) 21491712, 4096 00020000:00200000:3.0:1789760716.555865:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.555885:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5246, 5246] 00000008:00010000:3.0:1789760716.556076:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.556082:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.556094:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21487616, 21491712) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.556107:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21487616=0x147e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.556208:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.556211:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21491712 00000020:00010000:3.0:1789760716.556220:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.556224:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.556227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.556230:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.556233:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.556241:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.556248:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.556267:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760716.556395:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 21487616<>21491712 00000080:00200000:3.0:1789760716.630781:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5246, 5246] 00000080:00200000:3.0:1789760716.630814:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.630822:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.630840:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21491712 i_blocks 43015 00000020:00200000:3.0:1789760716.630844:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.630868:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.630913:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21491712, count: 4096 00000080:00200000:3.0:1789760716.630922:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21491712, bytes: 4096 00000080:00200000:3.0:1789760716.630938:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.630947:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.630950:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21491712, 21495808) 0 0 00000080:00200000:3.0:1789760716.630955:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.630972:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5247, 5247] 00020000:00200000:3.0:1789760716.630981:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21491712 chunk: [21491712, 21495808) 21495808, 4096 00020000:00200000:3.0:1789760716.630986:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.631007:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5247, 5247] 00000008:00010000:3.0:1789760716.631024:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.631030:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.631042:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21491712, 21495808) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.631056:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21491712=0x147f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.631132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.631136:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21495808 00000020:00010000:3.0:1789760716.631144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.631148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.631151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.631154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.631157:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.631166:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.631217:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.631238:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760716.631370:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 21491712<>21495808 00000080:00200000:3.0:1789760716.668581:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5247, 5247] 00000080:00200000:3.0:1789760716.668600:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.668607:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.668622:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21495808 i_blocks 43015 00000020:00200000:3.0:1789760716.668625:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.668635:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.668669:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21495808, count: 4096 00000080:00200000:3.0:1789760716.668677:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21495808, bytes: 4096 00000080:00200000:3.0:1789760716.668686:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.668695:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.668697:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21495808, 21499904) 0 0 00000080:00200000:3.0:1789760716.668702:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.668720:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5248, 5248] 00020000:00200000:3.0:1789760716.668729:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21495808 chunk: [21495808, 21499904) 21499904, 4096 00020000:00200000:3.0:1789760716.668734:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.668754:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5248, 5248] 00000008:00010000:3.0:1789760716.668973:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.668980:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.668994:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21495808, 21499904) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.669273:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21495808=0x1480000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.669349:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.669352:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21499904 00000020:00010000:3.0:1789760716.669361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.669365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.669368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.669372:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.669375:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.669383:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.669390:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.669411:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760716.669753:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 21495808<>21499904 00000100:00100000:3.0:1789760716.669978:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760716.670002:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495750144:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760716.776574:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000080:00200000:3.0:1789760716.846280:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5250, 5250] 00000080:00200000:3.0:1789760716.846297:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.846304:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.846320:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21508096 i_blocks 43015 00000020:00200000:3.0:1789760716.846324:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.846334:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.846368:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21508096, count: 4096 00000080:00200000:3.0:1789760716.846376:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21508096, bytes: 4096 00000080:00200000:3.0:1789760716.846391:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.846400:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.846402:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21508096, 21512192) 0 0 00000080:00200000:3.0:1789760716.846407:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.846425:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5251, 5251] 00020000:00200000:3.0:1789760716.846434:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21508096 chunk: [21508096, 21512192) 21512192, 4096 00020000:00200000:3.0:1789760716.846439:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.846458:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5251, 5251] 00000008:00010000:3.0:1789760716.846477:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.846483:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.846495:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21508096, 21512192) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.846509:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21508096=0x1483000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.846566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.846569:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21512192 00000020:00010000:3.0:1789760716.846577:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.846581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.846584:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.846588:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.846592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.846600:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.846607:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.846634:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760716.846855:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 21508096<>21512192 00000100:00100000:3.0:1789760716.846915:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760716.846934:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495750528:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760716.891829:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972568 gen 2 00000008:00010000:3.0:1789760716.891882:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760716.891891:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.891902:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.891928:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760716.891959:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495750528:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760716.936656:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5252, 5252] 00000080:00200000:3.0:1789760716.936674:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.936682:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.936699:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21516288 i_blocks 43015 00000020:00200000:3.0:1789760716.936703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.936724:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.936760:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21516288, count: 4096 00000080:00200000:3.0:1789760716.936767:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21516288, bytes: 4096 00000080:00200000:3.0:1789760716.936779:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.936788:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.936801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21516288, 21520384) 0 0 00000080:00200000:3.0:1789760716.936805:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.936824:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5253, 5253] 00020000:00200000:3.0:1789760716.936833:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21516288 chunk: [21516288, 21520384) 21520384, 4096 00020000:00200000:3.0:1789760716.936838:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.936859:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5253, 5253] 00000008:00010000:3.0:1789760716.936877:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.936883:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.936895:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21516288, 21520384) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.936911:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21516288=0x1485000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.936969:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.936972:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21520384 00000020:00010000:3.0:1789760716.936981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.936985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.936988:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.936992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.936995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.937004:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.937011:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.937028:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760716.937203:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 21516288<>21520384 00000080:00200000:3.0:1789760716.999345:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5253, 5253] 00000080:00200000:3.0:1789760716.999361:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760716.999367:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760716.999382:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21520384 i_blocks 43015 00000020:00200000:3.0:1789760716.999386:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760716.999395:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760716.999428:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21520384, count: 4096 00000080:00200000:3.0:1789760716.999436:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21520384, bytes: 4096 00000080:00200000:3.0:1789760716.999451:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760716.999459:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760716.999462:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21520384, 21524480) 0 0 00000080:00200000:3.0:1789760716.999466:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760716.999482:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5254, 5254] 00020000:00200000:3.0:1789760716.999491:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21520384 chunk: [21520384, 21524480) 21524480, 4096 00020000:00200000:3.0:1789760716.999495:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760716.999516:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5254, 5254] 00000008:00010000:3.0:1789760716.999535:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760716.999541:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760716.999554:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21520384, 21524480) append 0 append lockpos 0 00000080:00200000:3.0:1789760716.999568:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21520384=0x1486000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760716.999626:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760716.999629:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21524480 00000020:00010000:3.0:1789760716.999638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760716.999642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.999645:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760716.999649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760716.999652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760716.999659:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760716.999666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.999683:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760716.999823:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 21520384<>21524480 00000080:00200000:3.0:1789760717.032536:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5254, 5254] 00000080:00200000:3.0:1789760717.032554:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760717.032561:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760717.032577:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21524480 i_blocks 43015 00000020:00200000:3.0:1789760717.032581:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760717.032591:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760717.032629:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21524480, count: 4096 00000080:00200000:3.0:1789760717.032636:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21524480, bytes: 4096 00000080:00200000:3.0:1789760717.032649:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760717.032657:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760717.032659:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21524480, 21528576) 0 0 00000080:00200000:3.0:1789760717.032664:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760717.032682:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5255, 5255] 00020000:00200000:3.0:1789760717.032690:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21524480 chunk: [21524480, 21528576) 21528576, 4096 00020000:00200000:3.0:1789760717.032695:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760717.032787:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5255, 5255] 00000008:00010000:3.0:1789760717.032817:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760717.032822:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760717.032834:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21524480, 21528576) append 0 append lockpos 0 00000080:00200000:3.0:1789760717.032851:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21524480=0x1487000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760717.032912:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760717.032915:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21528576 00000020:00010000:3.0:1789760717.032923:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760717.032927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.032930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.032934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760717.032937:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760717.032945:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760717.032953:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760717.032969:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760717.033094:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 21524480<>21528576 00000080:00200000:3.0:1789760717.139895:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5255, 5255] 00000080:00200000:3.0:1789760717.139915:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760717.139921:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760717.139938:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21528576 i_blocks 43015 00000020:00200000:3.0:1789760717.139942:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760717.139952:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760717.139997:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21528576, count: 4096 00000080:00200000:3.0:1789760717.140004:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21528576, bytes: 4096 00000080:00200000:3.0:1789760717.140015:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760717.140024:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760717.140026:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21528576, 21532672) 0 0 00000080:00200000:3.0:1789760717.140031:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760717.140046:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5256, 5256] 00020000:00200000:3.0:1789760717.140055:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21528576 chunk: [21528576, 21532672) 21532672, 4096 00020000:00200000:3.0:1789760717.140060:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760717.140080:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5256, 5256] 00000008:00010000:3.0:1789760717.140097:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760717.140102:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760717.140113:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21528576, 21532672) append 0 append lockpos 0 00000080:00200000:3.0:1789760717.140127:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21528576=0x1488000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760717.140358:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760717.140361:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21532672 00000020:00010000:3.0:1789760717.140371:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760717.140375:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.140379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.140382:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760717.140386:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760717.140394:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760717.140403:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760717.140449:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760717.140583:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 21528576<>21532672 00000100:00100000:3.0:1789760717.140629:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760717.140646:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495751168:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.211067:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972573 gen 2 00000008:00010000:3.0:1789760717.211114:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:3.0:1789760717.211123:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.211132:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.211153:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:3.0:1789760717.211195:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495751168:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.375486:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495751552:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.474242:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972576 gen 2 00000008:00010000:3.0:1789760717.474301:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760717.474310:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.474324:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.474357:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760717.474393:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495751552:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.478639:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495751680:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760717.516685:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760717.516700:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.516711:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.516738:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760717.516766:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495751680:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.517590:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760717.517608:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495751808:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.561652:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972578 gen 2 00000008:00010000:3.0:1789760717.561717:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760717.561726:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.561737:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.561764:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760717.561806:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495751808:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.563063:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495751936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.610039:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972579 gen 2 00000008:00010000:3.0:1789760717.610090:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760717.610098:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.610108:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.610137:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760717.610209:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495751936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.610782:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495752064:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760717.656538:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760717.656554:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.656566:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.656589:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760717.656622:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495752064:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.657559:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495752192:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.691713:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972581 gen 2 00000008:00010000:3.0:1789760717.691795:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760717.691804:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.691815:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.691838:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760717.691875:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495752192:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.693391:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760717.693414:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495752320:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760717.737626:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760717.737643:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.737656:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.737680:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760717.737710:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495752320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.739428:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495752832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.783888:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972582 gen 2 00000100:00100000:3.0:1789760717.783927:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264e00 x1876699495752832/t4294972583(4294972583) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760733 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760717.783984:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760717.783991:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.784001:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.784023:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760717.784046:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495752832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.785478:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760717.785506:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495752960:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.837433:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972583 gen 2 00000100:00100000:3.0:1789760717.837475:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265880 x1876699495752960/t4294972584(4294972584) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760733 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760717.837525:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760717.837533:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.837543:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.837570:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760717.837594:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495752960:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.839143:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753088:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.882414:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972585 gen 2 00000008:00010000:3.0:1789760717.882477:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760717.882485:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.882496:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.882519:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760717.882549:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753088:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.883689:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760717.883711:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753216:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.981210:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972586 gen 2 00000008:00010000:3.0:1789760717.981262:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760717.981271:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.981294:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.981312:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760717.981342:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753216:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760717.982688:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753344:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760718.015438:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972587 gen 2 00000008:00010000:3.0:1789760718.015489:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760718.015499:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.015509:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.015536:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760718.015565:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753344:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760718.016394:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760718.016415:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753472:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760718.095261:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972588 gen 2 00000008:00010000:3.0:1789760718.095312:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760718.095321:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.095333:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.095436:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760718.095466:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753472:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760718.096578:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753600:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760718.127721:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972589 gen 2 00000008:00010000:3.0:1789760718.127776:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760718.127795:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.127813:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.127836:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760718.127866:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495753600:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760718.259436:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5275, 5275] 00000080:00200000:3.0:1789760718.259452:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760718.259459:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760718.259477:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21610496 i_blocks 43015 00000020:00200000:3.0:1789760718.259481:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760718.259491:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760718.259524:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21610496, count: 4096 00000080:00200000:3.0:1789760718.259531:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21610496, bytes: 4096 00000080:00200000:3.0:1789760718.259541:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760718.259549:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760718.259551:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21610496, 21614592) 0 0 00000080:00200000:3.0:1789760718.259556:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760718.259571:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5276, 5276] 00020000:00200000:3.0:1789760718.259579:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21610496 chunk: [21610496, 21614592) 21614592, 4096 00020000:00200000:3.0:1789760718.259584:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760718.259603:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5276, 5276] 00000008:00010000:3.0:1789760718.259620:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760718.259625:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760718.259634:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21610496, 21614592) append 0 append lockpos 0 00000080:00200000:3.0:1789760718.259647:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21610496=0x149c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760718.259706:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760718.259710:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21614592 00000020:00010000:3.0:1789760718.259718:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760718.259722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.259725:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.259728:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760718.259731:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760718.259738:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760718.259746:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760718.259928:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760718.260053:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 21610496<>21614592 00000080:00200000:3.0:1789760719.909886:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5299, 5299] 00000080:00200000:3.0:1789760719.909904:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760719.909910:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760719.909925:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21708800 i_blocks 43015 00000020:00200000:3.0:1789760719.909928:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760719.909938:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760719.909971:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21708800, count: 4096 00000080:00200000:3.0:1789760719.909978:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21708800, bytes: 4096 00000080:00200000:3.0:1789760719.909988:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760719.909997:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760719.910000:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21708800, 21712896) 0 0 00000080:00200000:3.0:1789760719.910004:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760719.910022:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5300, 5300] 00020000:00200000:3.0:1789760719.910031:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21708800 chunk: [21708800, 21712896) 21712896, 4096 00020000:00200000:3.0:1789760719.910037:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760719.910056:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5300, 5300] 00000008:00010000:3.0:1789760719.910075:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760719.910081:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760719.910094:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21708800, 21712896) append 0 append lockpos 0 00000080:00200000:3.0:1789760719.910108:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21708800=0x14b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760719.910196:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760719.910199:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21712896 00000020:00010000:3.0:1789760719.910207:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760719.910211:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.910214:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.910217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760719.910220:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760719.910228:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760719.910234:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760719.911136:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760719.913733:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 21708800<>21712896 00000100:00100000:3.0:1789760720.095554:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495757568:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.130063:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972620 gen 2 00000008:00010000:3.0:1789760720.130124:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760720.130132:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.130143:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.130207:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760720.130238:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495757568:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.131196:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760720.131216:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495757696:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760720.160635:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760720.160651:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.160662:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.160687:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760720.160716:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495757696:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.161645:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495757824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.278092:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972622 gen 2 00000008:00010000:3.0:1789760720.278160:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760720.278190:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.278202:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.278228:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760720.278262:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495757824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.279689:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760720.279711:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495757952:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760720.326582:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760720.326597:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.326609:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.326634:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760720.326662:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495757952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.327976:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.421912:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972624 gen 2 00000008:00010000:3.0:1789760720.421993:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760720.422002:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.422013:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.422036:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760720.422065:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.423387:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760720.423414:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.501156:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972625 gen 2 00000008:00010000:3.0:1789760720.501249:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760720.501258:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.501269:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.501293:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760720.501329:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.502883:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758336:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760720.605721:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760720.605737:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.605748:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.605951:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760720.605984:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.607726:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495758464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.683073:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972627 gen 2 00000008:00010000:3.0:1789760720.683149:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760720.683158:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.683199:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.683225:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760720.683257:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495758464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.750245:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758720:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.792342:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972629 gen 2 00000008:00010000:3.0:1789760720.792392:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760720.792401:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.792414:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.792446:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760720.792482:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758720:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.851193:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760720.851219:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758976:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760720.936407:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760720.936421:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.936434:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.936461:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760720.936500:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495758976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760720.937362:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495759104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760721.007232:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495759360:192.168.203.109@tcp:4:dd.0 00000020:00200000:3.0:1789760722.055530:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760722.055538:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21856256 00000020:00010000:3.0:1789760722.055552:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760722.055555:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.055559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d0b0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.055563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760722.055566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d0b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760722.055579:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760722.055587:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760722.055608:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760722.055766:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 21852160<>21856256 00000100:00100000:3.0:1789760722.055826:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760722.055863:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495761664:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760722.096280:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972652 gen 2 00000008:00010000:3.0:1789760722.096341:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:3.0:1789760722.096351:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.096362:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.096385:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:3.0:1789760722.096412:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495761664:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760722.164788:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495762048:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760722.267537:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5340, 5340] 00000080:00200000:3.0:1789760722.267554:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.267561:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.267576:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21876736 i_blocks 43015 00000020:00200000:3.0:1789760722.267580:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760722.267590:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760722.267624:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21876736, count: 4096 00000080:00200000:3.0:1789760722.267631:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21876736, bytes: 4096 00000080:00200000:3.0:1789760722.267646:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760722.267657:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760722.267659:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21876736, 21880832) 0 0 00000080:00200000:3.0:1789760722.267664:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760722.267682:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5341, 5341] 00020000:00200000:3.0:1789760722.267690:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21876736 chunk: [21876736, 21880832) 21880832, 4096 00020000:00200000:3.0:1789760722.267695:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760722.267715:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5341, 5341] 00000008:00010000:3.0:1789760722.267733:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760722.267738:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760722.267761:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21876736, 21880832) append 0 append lockpos 0 00000080:00200000:3.0:1789760722.267776:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21876736=0x14dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760722.267834:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760722.267837:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21880832 00000020:00010000:3.0:1789760722.267846:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760722.267850:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.267853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.267857:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760722.267860:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760722.267867:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760722.267874:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760722.267891:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760722.268020:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 21876736<>21880832 00000080:00200000:3.0:1789760722.308352:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5341, 5341] 00000080:00200000:3.0:1789760722.308371:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.308377:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760722.309296:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495762560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760722.776372:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000080:00200000:3.0:1789760722.826499:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5355, 5355] 00000080:00200000:3.0:1789760722.826515:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.826522:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.866794:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5356, 5356] 00000080:00200000:3.0:1789760722.866889:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.866898:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.866915:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21942272 i_blocks 43015 00000020:00200000:3.0:1789760722.866920:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760722.866930:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760722.866979:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21942272, count: 4096 00000080:00200000:3.0:1789760722.866987:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21942272, bytes: 4096 00000080:00200000:3.0:1789760722.867000:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760722.867009:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760722.867011:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21942272, 21946368) 0 0 00000080:00200000:3.0:1789760722.867016:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760722.867035:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5357, 5357] 00020000:00200000:3.0:1789760722.867044:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21942272 chunk: [21942272, 21946368) 21946368, 4096 00020000:00200000:3.0:1789760722.867049:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760722.867129:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5357, 5357] 00000008:00010000:3.0:1789760722.867149:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760722.867154:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760722.867273:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21942272, 21946368) append 0 append lockpos 0 00000080:00200000:3.0:1789760722.867292:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21942272=0x14ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760722.867362:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760722.867366:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21946368 00000020:00010000:3.0:1789760722.867375:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760722.867378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.867382:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.867385:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760722.867388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760722.867397:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760722.867405:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760722.867429:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760722.867566:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 21942272<>21946368 00000080:00200000:3.0:1789760722.895593:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5357, 5357] 00000080:00200000:3.0:1789760722.895612:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.895619:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.895637:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21946368 i_blocks 43015 00000020:00200000:3.0:1789760722.895640:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760722.895649:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760723.320248:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760723.321339:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495766272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760723.384933:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972685 gen 2 00000008:00010000:3.0:1789760723.384983:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760723.384992:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.385003:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.385144:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760723.385213:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495766272:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760723.433319:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5369, 5369] 00000080:00200000:3.0:1789760723.433363:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760723.433370:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760723.433385:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21995520 i_blocks 43015 00000020:00200000:3.0:1789760723.433389:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760723.433397:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760723.433442:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21995520, count: 4096 00000080:00200000:3.0:1789760723.433450:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21995520, bytes: 4096 00000080:00200000:3.0:1789760723.433462:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760723.433471:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760723.433474:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21995520, 21999616) 0 0 00000080:00200000:3.0:1789760723.433479:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760723.433497:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5370, 5370] 00020000:00200000:3.0:1789760723.433507:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21995520 chunk: [21995520, 21999616) 21999616, 4096 00020000:00200000:3.0:1789760723.433512:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760723.433532:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5370, 5370] 00000008:00010000:3.0:1789760723.433551:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760723.433557:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760723.433569:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21995520, 21999616) append 0 append lockpos 0 00000080:00200000:3.0:1789760723.433583:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=21995520=0x14fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760723.433643:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760723.433646:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21999616 00000020:00010000:3.0:1789760723.433656:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760723.433659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760723.433662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760723.433666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760723.433669:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760723.433676:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760723.433683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760723.433698:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760723.433845:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 21995520<>21999616 00000100:00100000:3.0:1789760723.433896:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495766528:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760723.487423:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972687 gen 2 00000008:00010000:3.0:1789760723.487475:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:3.0:1789760723.487485:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.487496:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.487520:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:3.0:1789760723.487551:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495766528:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760723.488495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495766656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760723.561256:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972688 gen 2 00000008:00010000:3.0:1789760723.561306:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760723.561315:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.561326:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.561350:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760723.561380:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495766656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760723.827355:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495767552:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760723.903872:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972695 gen 2 00000008:00010000:3.0:1789760723.903929:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760723.903938:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.903950:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.903983:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760723.904581:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495767552:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760724.017195:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5380, 5380] 00000080:00200000:3.0:1789760724.017211:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.017218:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.017236:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22040576 i_blocks 45063 00000020:00200000:3.0:1789760724.017240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.017250:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.017287:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22040576, count: 4096 00000080:00200000:3.0:1789760724.017294:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22040576, bytes: 4096 00000080:00200000:3.0:1789760724.017307:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.017317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.017319:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22040576, 22044672) 0 0 00000080:00200000:3.0:1789760724.017324:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.017340:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5381, 5381] 00020000:00200000:3.0:1789760724.017350:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22040576 chunk: [22040576, 22044672) 22044672, 4096 00020000:00200000:3.0:1789760724.017355:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.017376:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5381, 5381] 00000008:00010000:3.0:1789760724.017394:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.017400:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760724.017412:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22040576, 22044672) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.017429:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22040576=0x1505000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.017487:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.017490:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22044672 00000020:00010000:3.0:1789760724.017498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.017502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.017505:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.017509:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.017512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.017519:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.017526:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.017543:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760724.017666:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 22040576<>22044672 00000100:00100000:3.0:1789760724.017705:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760724.019367:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495767936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760724.059579:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972698 gen 2 00000008:00010000:3.0:1789760724.059636:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:3.0:1789760724.059646:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.059659:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.059688:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:3.0:1789760724.059721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495767936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760724.061708:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760724.061793:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495768064:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760724.141596:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972699 gen 2 00000008:00010000:3.0:1789760724.141639:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760724.141648:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.141657:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.141744:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760724.141773:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495768064:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760724.191307:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5383, 5383] 00000080:00200000:3.0:1789760724.191325:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.191331:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.191349:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22052864 i_blocks 45063 00000020:00200000:3.0:1789760724.191353:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.191363:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.191398:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22052864, count: 4096 00000080:00200000:3.0:1789760724.191406:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22052864, bytes: 4096 00000080:00200000:3.0:1789760724.191418:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.191426:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.191429:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22052864, 22056960) 0 0 00000080:00200000:3.0:1789760724.191433:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.191451:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5384, 5384] 00020000:00200000:3.0:1789760724.191459:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22052864 chunk: [22052864, 22056960) 22056960, 4096 00020000:00200000:3.0:1789760724.191464:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.191484:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5384, 5384] 00000008:00010000:3.0:1789760724.191502:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.191509:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760724.191520:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22052864, 22056960) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.191533:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22052864=0x1508000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.191590:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.191593:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22056960 00000020:00010000:3.0:1789760724.191602:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.191605:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.191609:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.191612:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.191615:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.191622:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.191629:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.191645:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760724.191846:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 22052864<>22056960 00000100:00100000:3.0:1789760725.642049:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495772288:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.700991:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972732 gen 2 00000008:00010000:3.0:1789760725.701030:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760725.701037:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.701046:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.701077:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760725.701102:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495772288:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.704138:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.748919:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972733 gen 2 00000008:00010000:3.0:1789760725.748954:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760725.748960:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.748969:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.748991:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760725.749015:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.749790:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760725.749807:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.799268:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972734 gen 2 00000008:00010000:3.0:1789760725.799322:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760725.799330:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.799344:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.799376:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760725.799407:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.800658:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772672:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.860463:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972735 gen 2 00000008:00010000:3.0:1789760725.860520:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760725.860529:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.860540:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.860567:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760725.860598:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772672:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.861624:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760725.861645:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772800:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760725.895309:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760725.895326:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.895338:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.895365:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760725.895395:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772800:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.896511:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772928:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.936142:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972736 gen 2 00000100:00100000:3.0:1789760725.936201:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264000 x1876699495772928/t4294972737(4294972737) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760741 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760725.936250:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760725.936257:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.936268:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.936363:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760725.936389:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495772928:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760725.937270:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495773056:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760726.003351:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972738 gen 2 00000008:00010000:3.0:1789760726.003416:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760726.003424:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.003433:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.003452:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760726.003483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495773056:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760726.565978:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495774464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760726.621401:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972749 gen 2 00000008:00010000:3.0:1789760726.621435:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760726.621442:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.621450:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.621472:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760726.621496:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495774464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760726.621986:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760726.622000:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495774592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760726.677664:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972750 gen 2 00000008:00010000:3.0:1789760726.677705:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760726.677711:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.677872:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.677946:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760726.677972:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495774592:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760726.746571:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5434, 5434] 00000080:00200000:3.0:1789760726.746583:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760726.746589:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.746602:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22261760 i_blocks 45063 00000020:00200000:3.0:1789760726.746605:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760726.746613:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760726.746639:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22261760, count: 4096 00000080:00200000:3.0:1789760726.746645:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22261760, bytes: 4096 00000080:00200000:3.0:1789760726.746652:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760726.746659:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760726.746661:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22261760, 22265856) 0 0 00000080:00200000:3.0:1789760726.746665:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760726.746678:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5435, 5435] 00020000:00200000:3.0:1789760726.746685:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22261760 chunk: [22261760, 22265856) 22265856, 4096 00020000:00200000:3.0:1789760726.746689:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760726.746703:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5435, 5435] 00000008:00010000:3.0:1789760726.746716:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760726.749640:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760726.749661:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22261760, 22265856) append 0 append lockpos 0 00000080:00200000:3.0:1789760726.749678:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22261760=0x153b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760726.749759:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760726.749763:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22265856 00000020:00010000:3.0:1789760726.749770:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760726.749772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.749775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.749779:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760726.749781:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760726.749790:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760726.749797:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760726.749832:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760726.749940:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 22261760<>22265856 00000100:00100000:3.0:1789760726.750001:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760726.750022:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495774848:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760726.853755:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:3.0:1789760726.853767:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.853776:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.853800:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:3.0:1789760726.853831:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495774848:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760726.901367:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5436, 5436] 00000080:00200000:3.0:1789760726.901380:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760726.901385:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.901398:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22269952 i_blocks 45063 00000020:00200000:3.0:1789760726.901402:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760726.901409:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760726.901437:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22269952, count: 4096 00000080:00200000:3.0:1789760726.901443:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22269952, bytes: 4096 00000080:00200000:3.0:1789760726.901451:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760726.901457:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760726.901459:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22269952, 22274048) 0 0 00000080:00200000:3.0:1789760726.901463:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760726.901476:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5437, 5437] 00020000:00200000:3.0:1789760726.901483:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22269952 chunk: [22269952, 22274048) 22274048, 4096 00020000:00200000:3.0:1789760726.901487:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760726.901513:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5437, 5437] 00000008:00010000:3.0:1789760726.901530:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760726.901538:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760726.901549:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22269952, 22274048) append 0 append lockpos 0 00000080:00200000:3.0:1789760726.901563:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22269952=0x153d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760726.901617:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760726.901621:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22274048 00000020:00010000:3.0:1789760726.901629:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760726.901633:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.901637:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.901640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760726.901644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760726.901651:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760726.901657:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760726.901676:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760726.901958:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 22269952<>22274048 00000080:00200000:3.0:1789760726.964764:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5437, 5437] 00000080:00200000:3.0:1789760726.964778:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760726.964783:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.964796:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22274048 i_blocks 45063 00000020:00200000:3.0:1789760726.964799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760726.964807:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760726.964832:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22274048, count: 4096 00000080:00200000:3.0:1789760726.964838:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22274048, bytes: 4096 00000080:00200000:3.0:1789760726.964845:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760726.964864:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760726.964867:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22274048, 22278144) 0 0 00000080:00200000:3.0:1789760726.964871:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760726.964885:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5438, 5438] 00020000:00200000:3.0:1789760726.964892:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22274048 chunk: [22274048, 22278144) 22278144, 4096 00020000:00200000:3.0:1789760726.964896:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760726.964913:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5438, 5438] 00000008:00010000:3.0:1789760726.964927:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760726.964934:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760726.964942:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22274048, 22278144) append 0 append lockpos 0 00000080:00200000:3.0:1789760726.964963:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22274048=0x153e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760726.965010:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760726.965013:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22278144 00000020:00010000:3.0:1789760726.965020:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760726.965023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.965026:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.965029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760726.965031:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760726.965705:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760726.965723:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760726.965746:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760726.965872:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 22274048<>22278144 00000080:00200000:3.0:1789760727.017696:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5438, 5438] 00000080:00200000:3.0:1789760727.017709:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.017723:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.017735:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22278144 i_blocks 45063 00000020:00200000:3.0:1789760727.017738:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.017750:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.017876:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22278144, count: 4096 00000080:00200000:3.0:1789760727.017883:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22278144, bytes: 4096 00000080:00200000:3.0:1789760727.017894:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.017900:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.017903:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22278144, 22282240) 0 0 00000080:00200000:3.0:1789760727.017907:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.017962:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5439, 5439] 00020000:00200000:3.0:1789760727.017970:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22278144 chunk: [22278144, 22282240) 22282240, 4096 00020000:00200000:3.0:1789760727.017973:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.017989:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5439, 5439] 00000008:00010000:3.0:1789760727.018003:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.018008:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.018018:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22278144, 22282240) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.018029:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22278144=0x153f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.018071:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.018073:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22282240 00000020:00010000:3.0:1789760727.018080:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.018083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.018086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.018089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.018091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.018097:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.018102:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.018116:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760727.018413:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 22278144<>22282240 00000080:00200000:3.0:1789760727.069866:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5439, 5439] 00000080:00200000:3.0:1789760727.069879:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.069884:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.069896:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22282240 i_blocks 45063 00000020:00200000:3.0:1789760727.069899:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.069907:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.069933:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22282240, count: 4096 00000080:00200000:3.0:1789760727.069939:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22282240, bytes: 4096 00000080:00200000:3.0:1789760727.070015:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.070023:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.070025:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22282240, 22286336) 0 0 00000080:00200000:3.0:1789760727.070029:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.070103:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5440, 5440] 00020000:00200000:3.0:1789760727.070110:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22282240 chunk: [22282240, 22286336) 22286336, 4096 00020000:00200000:3.0:1789760727.070115:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.070131:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5440, 5440] 00000008:00010000:3.0:1789760727.070145:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.070151:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.070160:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22282240, 22286336) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.070210:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22282240=0x1540000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.070256:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.070258:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22286336 00000020:00010000:3.0:1789760727.070266:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.070270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.070273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.070276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.070279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.070286:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.070291:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.070308:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760727.070409:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 22282240<>22286336 00000080:00200000:3.0:1789760727.139381:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5440, 5440] 00000080:00200000:3.0:1789760727.139398:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.139404:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.139418:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22286336 i_blocks 45063 00000020:00200000:3.0:1789760727.139423:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.139433:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.139458:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22286336, count: 4096 00000080:00200000:3.0:1789760727.139464:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22286336, bytes: 4096 00000080:00200000:3.0:1789760727.139473:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.139480:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.139482:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22286336, 22290432) 0 0 00000080:00200000:3.0:1789760727.139486:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.139499:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5441, 5441] 00020000:00200000:3.0:1789760727.139506:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22286336 chunk: [22286336, 22290432) 22290432, 4096 00020000:00200000:3.0:1789760727.139510:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.139525:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5441, 5441] 00000008:00010000:3.0:1789760727.139539:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.139545:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.139554:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22286336, 22290432) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.139566:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22286336=0x1541000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.139618:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.139621:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22290432 00000020:00010000:3.0:1789760727.139627:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.139630:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.139634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.139637:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.139639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.140623:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.140633:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.140661:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760727.141369:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 22286336<>22290432 00000080:00200000:3.0:1789760727.189973:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5441, 5441] 00000080:00200000:3.0:1789760727.189986:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.189999:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.190011:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22290432 i_blocks 45063 00000020:00200000:3.0:1789760727.190014:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.190023:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.190056:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22290432, count: 4096 00000080:00200000:3.0:1789760727.190062:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22290432, bytes: 4096 00000080:00200000:3.0:1789760727.190069:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.190076:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.190077:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22290432, 22294528) 0 0 00000080:00200000:3.0:1789760727.190081:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.190093:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5442, 5442] 00020000:00200000:3.0:1789760727.190099:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22290432 chunk: [22290432, 22294528) 22294528, 4096 00020000:00200000:3.0:1789760727.190103:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.190119:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5442, 5442] 00000008:00010000:3.0:1789760727.190133:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.190139:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.190148:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22290432, 22294528) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.190159:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22290432=0x1542000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.190228:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.190230:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22294528 00000020:00010000:3.0:1789760727.190237:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.190240:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.190243:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.190247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.190249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.190255:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.190260:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.190274:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760727.190374:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 22290432<>22294528 00000080:00200000:3.0:1789760727.247034:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5442, 5442] 00000080:00200000:3.0:1789760727.247057:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.247063:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.247074:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22294528 i_blocks 45063 00000020:00200000:3.0:1789760727.247077:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.247085:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.247108:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22294528, count: 4096 00000080:00200000:3.0:1789760727.247114:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22294528, bytes: 4096 00000080:00200000:3.0:1789760727.247121:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.247128:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.247130:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22294528, 22298624) 0 0 00000080:00200000:3.0:1789760727.247134:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.247148:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5443, 5443] 00020000:00200000:3.0:1789760727.247155:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22294528 chunk: [22294528, 22298624) 22298624, 4096 00020000:00200000:3.0:1789760727.247158:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.247212:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5443, 5443] 00000008:00010000:3.0:1789760727.247227:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.247232:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.247242:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22294528, 22298624) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.247252:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22294528=0x1543000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.247290:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.247292:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22298624 00000020:00010000:3.0:1789760727.247299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.247302:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.247305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.247308:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.247311:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.247317:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.247323:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.288022:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5443, 5443] 00000080:00200000:3.0:1789760727.288047:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.288054:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.288066:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22298624 i_blocks 45063 00000020:00200000:3.0:1789760727.288069:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.288078:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.288104:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22298624, count: 4096 00000080:00200000:3.0:1789760727.288112:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22298624, bytes: 4096 00000080:00200000:3.0:1789760727.288119:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.288127:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.288130:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22298624, 22302720) 0 0 00000080:00200000:3.0:1789760727.288134:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.288147:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5444, 5444] 00020000:00200000:3.0:1789760727.288155:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22298624 chunk: [22298624, 22302720) 22302720, 4096 00020000:00200000:3.0:1789760727.288159:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.288206:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5444, 5444] 00000008:00010000:3.0:1789760727.288220:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.288226:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.288235:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22298624, 22302720) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.288246:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22298624=0x1544000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.288285:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.288288:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22302720 00000020:00010000:3.0:1789760727.288294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.288297:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.288300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.288303:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.288306:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.288311:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.288317:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760727.290640:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495776000:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760727.328958:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5444, 5444] 00000080:00200000:3.0:1789760727.328971:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.328977:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.328990:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22302720 i_blocks 45063 00000020:00200000:3.0:1789760727.328993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.329001:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.329026:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22302720, count: 4096 00000080:00200000:3.0:1789760727.329032:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22302720, bytes: 4096 00000080:00200000:3.0:1789760727.329040:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.329057:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.329059:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22302720, 22306816) 0 0 00000080:00200000:3.0:1789760727.329063:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.329076:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5445, 5445] 00020000:00200000:3.0:1789760727.329083:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22302720 chunk: [22302720, 22306816) 22306816, 4096 00020000:00200000:3.0:1789760727.329087:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.329104:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5445, 5445] 00000008:00010000:3.0:1789760727.329119:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.329125:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.329134:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22302720, 22306816) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.329145:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22302720=0x1545000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.329228:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.329231:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22306816 00000020:00010000:3.0:1789760727.329238:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.329241:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.329244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.329247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.329250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.329256:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.329262:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.329276:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760727.329370:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 22302720<>22306816 00000080:00200000:3.0:1789760727.367543:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5445, 5445] 00000080:00200000:3.0:1789760727.367556:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.367562:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.367574:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22306816 i_blocks 45063 00000020:00200000:3.0:1789760727.367578:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.367586:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.367611:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22306816, count: 4096 00000080:00200000:3.0:1789760727.367617:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22306816, bytes: 4096 00000080:00200000:3.0:1789760727.367625:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.367633:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.367636:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22306816, 22310912) 0 0 00000080:00200000:3.0:1789760727.367640:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.367653:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5446, 5446] 00020000:00200000:3.0:1789760727.367660:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22306816 chunk: [22306816, 22310912) 22310912, 4096 00020000:00200000:3.0:1789760727.367664:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.367678:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5446, 5446] 00000008:00010000:3.0:1789760727.367692:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.367697:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.367705:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22306816, 22310912) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.367726:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22306816=0x1546000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.367774:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.367776:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22310912 00000020:00010000:3.0:1789760727.367783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.367786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.367789:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.367792:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.367795:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.367801:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.367807:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.367822:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760727.367915:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 22306816<>22310912 00000100:00100000:3.0:1789760727.367965:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760727.367979:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495776256:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760727.411165:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972763 gen 2 00000008:00010000:3.0:1789760727.411233:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:3.0:1789760727.411240:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.411248:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.411268:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:3.0:1789760727.411291:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495776256:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760727.464399:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5447, 5447] 00000080:00200000:3.0:1789760727.464413:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.464418:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.464431:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22315008 i_blocks 45063 00000020:00200000:3.0:1789760727.464434:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.464442:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.464469:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22315008, count: 4096 00000080:00200000:3.0:1789760727.464476:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22315008, bytes: 4096 00000080:00200000:3.0:1789760727.464484:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.464492:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.464494:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22315008, 22319104) 0 0 00000080:00200000:3.0:1789760727.464498:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.464511:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5448, 5448] 00020000:00200000:3.0:1789760727.464518:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22315008 chunk: [22315008, 22319104) 22319104, 4096 00020000:00200000:3.0:1789760727.464522:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.464539:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5448, 5448] 00000008:00010000:3.0:1789760727.464552:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.464557:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.464566:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22315008, 22319104) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.464576:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22315008=0x1548000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.464622:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.464625:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22319104 00000020:00010000:3.0:1789760727.464633:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.464636:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.464639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.464642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.464644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.464651:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.464657:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.464672:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760727.464782:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 22315008<>22319104 00000100:00100000:3.0:1789760727.464821:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760727.464835:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495776512:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760727.506310:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972765 gen 2 00000008:00010000:3.0:1789760727.506359:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:3.0:1789760727.506367:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.506375:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.506447:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:3.0:1789760727.506476:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495776512:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760727.507117:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495776640:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760727.562686:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972766 gen 2 00000008:00010000:3.0:1789760727.562745:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760727.562752:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.562762:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.562780:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760727.562803:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495776640:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760727.951762:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5457, 5457] 00000080:00200000:3.0:1789760727.951776:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.951781:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.951792:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22355968 i_blocks 45063 00000020:00200000:3.0:1789760727.951795:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.951803:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.951831:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22355968, count: 4096 00000080:00200000:3.0:1789760727.951837:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22355968, bytes: 4096 00000080:00200000:3.0:1789760727.951846:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.951864:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.951866:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22355968, 22360064) 0 0 00000080:00200000:3.0:1789760727.951870:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.951883:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5458, 5458] 00020000:00200000:3.0:1789760727.951890:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22355968 chunk: [22355968, 22360064) 22360064, 4096 00020000:00200000:3.0:1789760727.951894:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.951910:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5458, 5458] 00000008:00010000:3.0:1789760727.951922:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.951928:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.951936:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22355968, 22360064) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.951946:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22355968=0x1552000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.951993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.951997:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22360064 00000020:00010000:3.0:1789760727.952004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.952007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.952009:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.952012:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.952014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.952021:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.952026:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.952049:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760727.952140:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 22355968<>22360064 00000080:00200000:3.0:1789760727.991988:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5458, 5458] 00000080:00200000:3.0:1789760727.992001:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.992006:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.992017:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22360064 i_blocks 45063 00000020:00200000:3.0:1789760727.992021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.992029:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.992111:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22360064, count: 4096 00000080:00200000:3.0:1789760727.992119:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22360064, bytes: 4096 00000080:00200000:3.0:1789760727.992126:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.992132:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.992134:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22360064, 22364160) 0 0 00000080:00200000:3.0:1789760727.992138:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.992152:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5459, 5459] 00020000:00200000:3.0:1789760727.992159:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22360064 chunk: [22360064, 22364160) 22364160, 4096 00020000:00200000:3.0:1789760727.992163:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.992264:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5459, 5459] 00000008:00010000:3.0:1789760727.992279:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.992284:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760727.992293:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22360064, 22364160) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.992304:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22360064=0x1553000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.992340:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.992343:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22364160 00000020:00010000:3.0:1789760727.992418:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.992422:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.992425:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.992428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.992430:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.992437:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.992444:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.060928:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5459, 5459] 00000080:00200000:3.0:1789760728.060941:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.060946:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.060958:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22364160 i_blocks 45063 00000020:00200000:3.0:1789760728.060961:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.060970:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.060995:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22364160, count: 4096 00000080:00200000:3.0:1789760728.061000:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22364160, bytes: 4096 00000080:00200000:3.0:1789760728.061010:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.061017:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.061019:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22364160, 22368256) 0 0 00000080:00200000:3.0:1789760728.061022:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.061035:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5460, 5460] 00020000:00200000:3.0:1789760728.061042:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22364160 chunk: [22364160, 22368256) 22368256, 4096 00020000:00200000:3.0:1789760728.061047:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.061062:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5460, 5460] 00000008:00010000:3.0:1789760728.061074:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.061080:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760728.061089:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22364160, 22368256) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.061100:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22364160=0x1554000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.061145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.061148:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22368256 00000020:00010000:3.0:1789760728.061155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.061159:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.061161:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.061164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.063485:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.063508:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.063517:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.063544:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760728.063653:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 22364160<>22368256 00000080:00200000:3.0:1789760728.100612:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5460, 5460] 00000080:00200000:3.0:1789760728.100624:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.100630:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.100642:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22368256 i_blocks 45063 00000020:00200000:3.0:1789760728.100646:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.100654:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.100680:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22368256, count: 4096 00000080:00200000:3.0:1789760728.100686:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22368256, bytes: 4096 00000080:00200000:3.0:1789760728.100705:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.100723:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.100725:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22368256, 22372352) 0 0 00000080:00200000:3.0:1789760728.100729:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.100741:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5461, 5461] 00020000:00200000:3.0:1789760728.100749:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22368256 chunk: [22368256, 22372352) 22372352, 4096 00020000:00200000:3.0:1789760728.100753:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.100769:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5461, 5461] 00000008:00010000:3.0:1789760728.100780:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.100786:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760728.100794:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22368256, 22372352) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.100805:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22368256=0x1555000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.100850:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.100853:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22372352 00000020:00010000:3.0:1789760728.100860:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.100864:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.100866:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.100869:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.100872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.100877:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.100883:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.100897:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760728.100990:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 22368256<>22372352 00000080:00200000:3.0:1789760728.141277:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5461, 5461] 00000080:00200000:3.0:1789760728.141289:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.141294:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.141306:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22372352 i_blocks 45063 00000020:00200000:3.0:1789760728.141309:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.141317:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.141341:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22372352, count: 4096 00000080:00200000:3.0:1789760728.141348:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22372352, bytes: 4096 00000080:00200000:3.0:1789760728.141355:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.141363:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.141365:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22372352, 22376448) 0 0 00000080:00200000:3.0:1789760728.141368:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.141381:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5462, 5462] 00020000:00200000:3.0:1789760728.141387:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22372352 chunk: [22372352, 22376448) 22376448, 4096 00020000:00200000:3.0:1789760728.141391:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.141406:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5462, 5462] 00000008:00010000:3.0:1789760728.141418:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.141424:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760728.141431:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22372352, 22376448) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.141441:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22372352=0x1556000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.141484:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.141487:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22376448 00000020:00010000:3.0:1789760728.141494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.141498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.141501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.141504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.141506:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.141513:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.141519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.141532:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760728.141620:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 22372352<>22376448 00000080:00200000:3.0:1789760728.222764:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5462, 5462] 00000080:00200000:3.0:1789760728.222777:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.222783:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.222796:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22376448 i_blocks 45063 00000020:00200000:3.0:1789760728.222799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.222808:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.222833:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22376448, count: 4096 00000080:00200000:3.0:1789760728.222840:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22376448, bytes: 4096 00000080:00200000:3.0:1789760728.222847:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.222856:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.222858:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22376448, 22380544) 0 0 00000080:00200000:3.0:1789760728.222862:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.222875:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5463, 5463] 00020000:00200000:3.0:1789760728.222882:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22376448 chunk: [22376448, 22380544) 22380544, 4096 00020000:00200000:3.0:1789760728.222886:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.222902:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5463, 5463] 00000008:00010000:3.0:1789760728.222915:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.222921:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760728.222929:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22376448, 22380544) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.222942:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22376448=0x1557000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.222995:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.222998:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22380544 00000020:00010000:3.0:1789760728.223004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.223008:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.223011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.223015:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.223018:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.223025:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.223043:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.223058:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760728.223604:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 22376448<>22380544 00000100:00100000:3.0:1789760728.223650:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495778816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.296762:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972780 gen 2 00000008:00010000:3.0:1789760728.296799:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760728.296805:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.296814:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.296834:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760728.296856:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495778816:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760728.498426:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5467, 5467] 00000080:00200000:3.0:1789760728.498442:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.498447:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.498460:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22396928 i_blocks 45063 00000020:00200000:3.0:1789760728.498466:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000100:00100000:3.0:1789760728.548157:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760728.548203:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495779584:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.591526:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972786 gen 2 00000008:00010000:3.0:1789760728.591561:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760728.591566:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.591575:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.591598:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760728.591622:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495779584:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.592283:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495779712:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760728.627442:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760728.627452:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.627461:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.627481:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760728.627502:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495779712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.628660:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760728.628679:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495779840:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.672390:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972788 gen 2 00000008:00010000:3.0:1789760728.672439:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760728.672446:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.672455:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.672479:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760728.672513:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495779840:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.674060:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495779968:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.716231:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972789 gen 2 00000008:00010000:3.0:1789760728.716267:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760728.716273:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.716283:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.716306:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760728.716330:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495779968:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.717333:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760728.717349:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495780096:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.791693:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972790 gen 2 00000008:00010000:3.0:1789760728.791747:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760728.791753:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.791764:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.791849:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760728.791874:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495780096:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760728.840013:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22425600 i_blocks 45063 00000020:00200000:3.0:1789760728.840020:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.840043:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.840071:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22425600, count: 4096 00000080:00200000:3.0:1789760728.840078:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22425600, bytes: 4096 00000080:00200000:3.0:1789760728.840090:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.840097:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.840099:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22425600, 22429696) 0 0 00000080:00200000:3.0:1789760728.840102:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.840115:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5475, 5475] 00020000:00200000:3.0:1789760728.840122:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22425600 chunk: [22425600, 22429696) 22429696, 4096 00020000:00200000:3.0:1789760728.840126:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.840144:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5475, 5475] 00000008:00010000:3.0:1789760728.840159:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.840186:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760728.840196:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22425600, 22429696) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.840207:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22425600=0x1563000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.840263:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.840266:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22429696 00000020:00010000:3.0:1789760728.840272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.840275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.840278:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.840281:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.840284:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.840291:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.840297:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.840314:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760728.840428:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 22425600<>22429696 00000100:00100000:3.0:1789760728.840472:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760728.840493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495780352:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.872800:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972792 gen 2 00000008:00010000:3.0:1789760728.872835:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760728.872842:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.872850:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.872873:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760728.872901:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495780352:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.873726:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495780480:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.912365:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972793 gen 2 00000008:00010000:3.0:1789760728.912403:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760728.912410:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.912419:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.912483:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760728.912509:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495780480:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.913555:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495780608:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.955866:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972794 gen 2 00000008:00010000:3.0:1789760728.955902:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760728.955908:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.955915:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.955993:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760728.956030:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495780608:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760728.956697:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495780736:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.002016:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972795 gen 2 00000008:00010000:3.0:1789760729.002053:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760729.002059:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.002067:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.002092:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760729.002114:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495780736:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.003154:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495780864:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760729.043925:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760729.043939:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.043948:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.043971:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760729.043996:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495780864:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.044794:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760729.044808:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495780992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.083891:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972797 gen 2 00000008:00010000:3.0:1789760729.083939:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760729.083946:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.083955:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.083978:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760729.084001:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495780992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.085293:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495781120:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760729.128254:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760729.128265:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.128275:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.128296:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760729.128320:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495781120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.129223:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760729.129239:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495781248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.168282:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972799 gen 2 00000008:00010000:3.0:1789760729.168331:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760729.168337:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.168347:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.168367:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760729.168392:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495781248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.169114:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495781376:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.206424:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972800 gen 2 00000008:00010000:3.0:1789760729.206474:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760729.206483:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.206496:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.206586:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760729.206617:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495781376:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760729.245038:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5484, 5484] 00000080:00200000:3.0:1789760729.245054:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.245106:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.245123:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22466560 i_blocks 45063 00000020:00200000:3.0:1789760729.245127:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.245136:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.245325:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22466560, count: 4096 00000080:00200000:3.0:1789760729.245333:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22466560, bytes: 4096 00000080:00200000:3.0:1789760729.245535:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.245544:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.245547:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22466560, 22470656) 0 0 00000080:00200000:3.0:1789760729.245552:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.245569:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5485, 5485] 00020000:00200000:3.0:1789760729.245576:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22466560 chunk: [22466560, 22470656) 22470656, 4096 00020000:00200000:3.0:1789760729.245579:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.245598:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5485, 5485] 00000008:00010000:3.0:1789760729.245611:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.245616:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760729.245625:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22466560, 22470656) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.245636:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22466560=0x156d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.245696:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.245705:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22470656 00000020:00010000:3.0:1789760729.245714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.245717:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.245720:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.245724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.245727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.245734:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.245741:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.245761:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760729.245893:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 22466560<>22470656 00000100:00100000:3.0:1789760729.245942:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760729.245961:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495781632:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760729.282228:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760729.282246:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.282273:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.282295:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760729.282328:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495781632:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760729.333742:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5486, 5486] 00000080:00200000:3.0:1789760729.333759:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.333768:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.333784:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22474752 i_blocks 45063 00000020:00200000:3.0:1789760729.333787:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.333796:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.333827:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22474752, count: 4096 00000080:00200000:3.0:1789760729.333833:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22474752, bytes: 4096 00000080:00200000:3.0:1789760729.333843:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.333851:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.333854:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22474752, 22478848) 0 0 00000080:00200000:3.0:1789760729.333858:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.333875:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5487, 5487] 00020000:00200000:3.0:1789760729.333883:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22474752 chunk: [22474752, 22478848) 22478848, 4096 00020000:00200000:3.0:1789760729.333888:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.333908:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5487, 5487] 00000008:00010000:3.0:1789760729.333926:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.333932:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760729.333943:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22474752, 22478848) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.333956:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22474752=0x156f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.334007:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.334068:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22478848 00000020:00010000:3.0:1789760729.334078:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.334083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.334086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.334089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.334092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.334099:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.334106:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.334121:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760729.334292:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 22474752<>22478848 00000100:00100000:3.0:1789760729.334335:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760729.334353:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495781888:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760729.374453:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760729.374468:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.374480:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.374562:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760729.374592:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495781888:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.493409:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495782272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.546446:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972807 gen 2 00000008:00010000:3.0:1789760729.546513:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760729.546521:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.546532:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.546562:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760729.546663:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495782272:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760729.586150:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5491, 5491] 00000080:00200000:3.0:1789760729.586200:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.586207:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.586225:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22495232 i_blocks 45063 00000020:00200000:3.0:1789760729.586229:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.586253:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.586289:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22495232, count: 4096 00000080:00200000:3.0:1789760729.586297:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22495232, bytes: 4096 00000080:00200000:3.0:1789760729.586306:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.586315:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.586317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22495232, 22499328) 0 0 00000080:00200000:3.0:1789760729.586322:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.586339:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5492, 5492] 00020000:00200000:3.0:1789760729.586347:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22495232 chunk: [22495232, 22499328) 22499328, 4096 00020000:00200000:3.0:1789760729.586352:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.586372:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5492, 5492] 00000008:00010000:3.0:1789760729.586391:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.586396:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760729.586408:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22495232, 22499328) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.586422:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22495232=0x1574000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.586476:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.586479:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22499328 00000020:00010000:3.0:1789760729.586487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.586491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.586494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.586498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.586501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.586509:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.586516:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.586531:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760729.586660:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 22495232<>22499328 00000100:00100000:3.0:1789760729.586717:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760729.586735:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495782528:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760729.621334:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760729.621351:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.621362:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.621392:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760729.621422:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495782528:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760729.654319:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5493, 5493] 00000080:00200000:3.0:1789760729.654339:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.654345:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.654362:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22503424 i_blocks 45063 00000020:00200000:3.0:1789760729.654365:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.654375:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.654410:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22503424, count: 4096 00000080:00200000:3.0:1789760729.654418:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22503424, bytes: 4096 00000080:00200000:3.0:1789760729.654431:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.654439:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.654441:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22503424, 22507520) 0 0 00000080:00200000:3.0:1789760729.654445:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.654460:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5494, 5494] 00020000:00200000:3.0:1789760729.654469:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22503424 chunk: [22503424, 22507520) 22507520, 4096 00020000:00200000:3.0:1789760729.654473:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.654493:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5494, 5494] 00000008:00010000:3.0:1789760729.654510:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.654516:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760729.654528:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22503424, 22507520) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.654541:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22503424=0x1576000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.654594:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.654597:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22507520 00000020:00010000:3.0:1789760729.654605:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.654608:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.654611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.654615:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.654618:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.654626:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.654633:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.654650:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760729.654805:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 22503424<>22507520 00000080:00200000:3.0:1789760729.718750:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5494, 5494] 00000080:00200000:3.0:1789760729.718771:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.718779:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.718796:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22507520 i_blocks 45063 00000020:00200000:3.0:1789760729.718799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.718810:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.718845:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22507520, count: 4096 00000080:00200000:3.0:1789760729.718853:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22507520, bytes: 4096 00000080:00200000:3.0:1789760729.718866:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.718875:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.718878:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22507520, 22511616) 0 0 00000080:00200000:3.0:1789760729.718882:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.718909:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5495, 5495] 00020000:00200000:3.0:1789760729.718917:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22507520 chunk: [22507520, 22511616) 22511616, 4096 00020000:00200000:3.0:1789760729.718922:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.718948:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5495, 5495] 00000008:00010000:3.0:1789760729.718967:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.718972:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760729.718984:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22507520, 22511616) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.718999:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22507520=0x1577000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.719449:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.719454:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22511616 00000020:00010000:3.0:1789760729.719464:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.719468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.719471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.719475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.719479:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.719486:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.719494:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.719513:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760729.719634:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 22507520<>22511616 00000100:00100000:3.0:1789760729.719687:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760729.719715:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495782912:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.760738:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972812 gen 2 00000008:00010000:3.0:1789760729.760787:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:3.0:1789760729.760796:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.760807:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.760834:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:3.0:1789760729.760875:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495782912:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760729.802234:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5496, 5496] 00000080:00200000:3.0:1789760729.802252:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.802258:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.802275:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22515712 i_blocks 45063 00000020:00200000:3.0:1789760729.802278:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.802288:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.802321:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22515712, count: 4096 00000080:00200000:3.0:1789760729.802327:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22515712, bytes: 4096 00000080:00200000:3.0:1789760729.802338:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.802347:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.802349:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22515712, 22519808) 0 0 00000080:00200000:3.0:1789760729.802354:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.802370:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5497, 5497] 00020000:00200000:3.0:1789760729.802379:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22515712 chunk: [22515712, 22519808) 22519808, 4096 00020000:00200000:3.0:1789760729.802383:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.802403:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5497, 5497] 00000008:00010000:3.0:1789760729.802420:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.802426:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760729.802438:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22515712, 22519808) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.802453:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22515712=0x1579000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.802512:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.802515:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22519808 00000020:00010000:3.0:1789760729.802523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.802526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.802530:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.802533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.802536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.802543:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.802551:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.802567:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760729.802690:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 22515712<>22519808 00000080:00200000:3.0:1789760729.836682:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5497, 5497] 00000080:00200000:3.0:1789760729.836708:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.836715:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.836730:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22519808 i_blocks 45063 00000020:00200000:3.0:1789760729.836733:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.836743:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.836781:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22519808, count: 4096 00000080:00200000:3.0:1789760729.836789:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22519808, bytes: 4096 00000080:00200000:3.0:1789760729.836801:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.836810:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.836812:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22519808, 22523904) 0 0 00000080:00200000:3.0:1789760729.836816:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.836833:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5498, 5498] 00020000:00200000:3.0:1789760729.836842:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22519808 chunk: [22519808, 22523904) 22523904, 4096 00020000:00200000:3.0:1789760729.836859:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.836882:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5498, 5498] 00000008:00010000:3.0:1789760729.836898:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.836903:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760729.836914:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22519808, 22523904) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.836927:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22519808=0x157a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.836986:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.836988:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22523904 00000020:00010000:3.0:1789760729.836997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.837000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.837004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.837008:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.837011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.837130:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.837138:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.837155:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760729.837397:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 22519808<>22523904 00000100:00100000:3.0:1789760729.837442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495783296:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760729.871265:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:3.0:1789760729.871281:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.871293:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.871372:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:3.0:1789760729.871403:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495783296:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760729.872514:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495783424:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.239981:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495784320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.293548:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972823 gen 2 00000008:00010000:3.0:1789760730.293598:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760730.293606:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.293618:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.293644:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760730.293675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495784320:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760730.580716:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5513, 5513] 00000080:00200000:3.0:1789760730.580734:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.580741:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760730.580757:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22585344 i_blocks 45063 00000020:00200000:3.0:1789760730.580760:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760730.580770:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760730.580802:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22585344, count: 4096 00000080:00200000:3.0:1789760730.580809:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22585344, bytes: 4096 00000080:00200000:3.0:1789760730.580820:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760730.580830:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760730.580833:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22585344, 22589440) 0 0 00000080:00200000:3.0:1789760730.580837:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760730.580861:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5514, 5514] 00020000:00200000:3.0:1789760730.580869:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22585344 chunk: [22585344, 22589440) 22589440, 4096 00020000:00200000:3.0:1789760730.580874:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760730.580894:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5514, 5514] 00000008:00010000:3.0:1789760730.580912:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760730.580919:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760730.580933:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22585344, 22589440) append 0 append lockpos 0 00000080:00200000:3.0:1789760730.580947:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22585344=0x158a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760730.581009:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760730.581012:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22589440 00000020:00010000:3.0:1789760730.581021:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760730.581024:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.581027:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.581031:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760730.581034:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760730.581042:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760730.581050:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760730.581343:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760730.581476:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 22585344<>22589440 00000080:00200000:3.0:1789760730.620321:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5514, 5514] 00000080:00200000:3.0:1789760730.620339:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.620344:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760730.621623:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495785472:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.665421:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972832 gen 2 00000008:00010000:3.0:1789760730.665474:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760730.665482:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.665493:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.665523:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760730.665555:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495785472:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.667279:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495785600:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760730.711572:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760730.711589:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.711603:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.711634:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760730.711677:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495785600:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.712570:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760730.712589:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495785728:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.748520:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972833 gen 2 00000100:00100000:3.0:1789760730.748565:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495785728/t4294972834(4294972834) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760746 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760730.748614:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760730.748622:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.748634:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.748660:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760730.748686:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495785728:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.750104:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760730.750127:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495785856:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.780323:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972835 gen 2 00000008:00010000:3.0:1789760730.780370:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760730.780377:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.780386:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.780409:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760730.780439:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495785856:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.781386:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495785984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.844358:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972836 gen 2 00000008:00010000:3.0:1789760730.844412:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760730.844421:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.844433:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.844457:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760730.844752:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495785984:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760730.845338:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760730.845344:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760730.845360:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760730.845457:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 22609920<>22614016 00000100:00100000:3.0:1789760730.845798:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495786112:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760730.887677:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972837 gen 2 00000008:00010000:3.0:1789760730.887864:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760730.887875:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.887886:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.887980:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760730.888012:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495786112:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760730.888108:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5520, 5520] 00000080:00200000:2.0:1789760730.888127:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760730.888134:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760730.888150:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22614016 i_blocks 45063 00000020:00200000:2.0:1789760730.888314:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760730.888326:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760730.888365:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22614016, count: 4096 00000080:00200000:2.0:1789760730.888373:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22614016, bytes: 4096 00000080:00200000:2.0:1789760730.888389:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760730.888399:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760730.888402:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22614016, 22618112) 0 0 00000080:00200000:2.0:1789760730.888407:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760730.888427:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5521, 5521] 00020000:00200000:2.0:1789760730.888436:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22614016 chunk: [22614016, 22618112) 22618112, 4096 00020000:00200000:2.0:1789760730.888442:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760730.888465:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5521, 5521] 00000008:00010000:2.0:1789760730.888709:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760730.888717:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760730.888736:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22614016, 22618112) append 0 append lockpos 0 00000080:00200000:2.0:1789760730.888828:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22614016=0x1591000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760730.888898:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760730.888902:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22618112 00000020:00010000:2.0:1789760730.888912:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760730.888916:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.888919:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.888923:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760730.888926:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760730.888936:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760730.888946:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760730.888973:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760730.889114:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 22614016<>22618112 00000100:00100000:3.0:1789760730.889214:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495786240:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760730.929249:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972838 gen 2 00000008:00010000:2.0:1789760730.929302:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760730.929310:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760730.929321:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760730.929350:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760730.929385:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495786240:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.191451:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5545, 5545] 00000080:00200000:2.0:1789760732.191466:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.191471:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.191484:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22716416 i_blocks 45063 00000020:00200000:2.0:1789760732.191488:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.191497:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.191534:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22716416, count: 4096 00000080:00200000:2.0:1789760732.191540:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22716416, bytes: 4096 00000080:00200000:2.0:1789760732.191549:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.191556:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.191558:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22716416, 22720512) 0 0 00000080:00200000:2.0:1789760732.191562:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.191574:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5546, 5546] 00020000:00200000:2.0:1789760732.191582:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22716416 chunk: [22716416, 22720512) 22720512, 4096 00020000:00200000:2.0:1789760732.191586:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.191603:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5546, 5546] 00000008:00010000:2.0:1789760732.191617:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.191622:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.191632:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22716416, 22720512) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.191642:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22716416=0x15aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.191833:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.191837:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22720512 00000020:00010000:2.0:1789760732.191845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.191848:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.191851:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.191854:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.191857:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.191863:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.191869:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.191883:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760732.191974:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 22716416<>22720512 00000100:00100000:2.0:1789760732.192016:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760732.192030:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495789440:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760732.230652:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972862 gen 2 00000100:00100000:2.0:1789760732.230694:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266d80 x1876699495789440/t4294972863(4294972863) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760748 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760732.230857:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760732.230864:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.231007:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.231033:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760732.231054:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495789440:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.261695:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5547, 5547] 00000080:00200000:2.0:1789760732.261710:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.261715:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.261737:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22724608 i_blocks 45063 00000020:00200000:2.0:1789760732.261740:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.261749:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.261778:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22724608, count: 4096 00000080:00200000:2.0:1789760732.261784:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22724608, bytes: 4096 00000080:00200000:2.0:1789760732.261792:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.261799:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.261801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22724608, 22728704) 0 0 00000080:00200000:2.0:1789760732.261804:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.261819:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5548, 5548] 00020000:00200000:2.0:1789760732.261826:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22724608 chunk: [22724608, 22728704) 22728704, 4096 00020000:00200000:2.0:1789760732.261830:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.261847:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5548, 5548] 00000008:00010000:2.0:1789760732.261863:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.261869:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.261879:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22724608, 22728704) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.261890:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22724608=0x15ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.261937:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.261940:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22728704 00000020:00010000:2.0:1789760732.261947:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.261950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.261953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.261955:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.261958:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.261965:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.261970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.261985:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760732.262083:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 22724608<>22728704 00000100:00100000:3.0:1789760732.262255:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760732.262271:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495789696:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760732.316692:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972864 gen 2 00000100:00100000:3.0:1789760732.316722:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265f80 x1876699495789696/t4294972865(4294972865) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760748 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760732.316758:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760732.316764:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.316773:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.316793:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760732.316813:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495789696:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.317008:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5548, 5548] 00000080:00200000:2.0:1789760732.317030:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.317036:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.317048:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22728704 i_blocks 45063 00000020:00200000:2.0:1789760732.317052:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.317060:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.317085:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22728704, count: 4096 00000080:00200000:2.0:1789760732.317091:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22728704, bytes: 4096 00000080:00200000:2.0:1789760732.317099:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.317105:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.317108:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22728704, 22732800) 0 0 00000080:00200000:2.0:1789760732.317111:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.317124:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5549, 5549] 00020000:00200000:2.0:1789760732.317131:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22728704 chunk: [22728704, 22732800) 22732800, 4096 00020000:00200000:2.0:1789760732.317135:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.317149:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5549, 5549] 00000008:00010000:2.0:1789760732.317163:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.317192:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.317202:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22728704, 22732800) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.317212:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22728704=0x15ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.317257:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.317260:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22732800 00000020:00010000:2.0:1789760732.317267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.317270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.317273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.317275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.317278:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.317284:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.317289:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.317303:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760732.317408:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 22728704<>22732800 00000100:00100000:3.0:1789760732.317672:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495789824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760732.385124:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972866 gen 2 00000008:00010000:3.0:1789760732.385203:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760732.385210:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.385219:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.385242:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760732.385264:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495789824:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.385356:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5549, 5549] 00000080:00200000:2.0:1789760732.385368:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.385373:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.385385:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22732800 i_blocks 45063 00000020:00200000:2.0:1789760732.385388:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.385396:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.385421:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22732800, count: 4096 00000080:00200000:2.0:1789760732.385428:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22732800, bytes: 4096 00000080:00200000:2.0:1789760732.385435:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.385443:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.385445:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22732800, 22736896) 0 0 00000080:00200000:2.0:1789760732.385449:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.385461:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5550, 5550] 00020000:00200000:2.0:1789760732.385468:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22732800 chunk: [22732800, 22736896) 22736896, 4096 00020000:00200000:2.0:1789760732.385472:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.385487:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5550, 5550] 00000008:00010000:2.0:1789760732.385502:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.385508:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.385517:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22732800, 22736896) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.385529:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22732800=0x15ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.385586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.385589:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22736896 00000020:00010000:2.0:1789760732.385596:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.385599:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.385602:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.385605:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.385607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.385613:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.385617:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.385630:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760732.385735:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 22732800<>22736896 00000100:00100000:3.0:1789760732.386811:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760732.386832:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495789952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760732.435818:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972867 gen 2 00000008:00010000:3.0:1789760732.435855:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760732.435862:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.435872:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.435893:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760732.435916:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495789952:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.436023:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5550, 5550] 00000080:00200000:2.0:1789760732.436037:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.436043:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.436055:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22736896 i_blocks 45063 00000020:00200000:2.0:1789760732.436058:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.436066:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.436091:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22736896, count: 4096 00000080:00200000:2.0:1789760732.436097:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22736896, bytes: 4096 00000080:00200000:2.0:1789760732.436105:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.436112:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.436114:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22736896, 22740992) 0 0 00000080:00200000:2.0:1789760732.436118:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.436130:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5551, 5551] 00020000:00200000:2.0:1789760732.436137:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22736896 chunk: [22736896, 22740992) 22740992, 4096 00020000:00200000:2.0:1789760732.436141:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.436157:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5551, 5551] 00000008:00010000:2.0:1789760732.436211:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.436216:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.436225:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22736896, 22740992) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.436235:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22736896=0x15af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.436277:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.436280:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22740992 00000020:00010000:2.0:1789760732.436286:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.436288:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.436291:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.436294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.436296:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.436302:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.436307:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.436320:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760732.436412:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 22736896<>22740992 00000100:00100000:3.0:1789760732.436570:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495790080:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760732.533748:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760732.533759:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.533769:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.533788:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760732.533813:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495790080:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.533967:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5551, 5551] 00000080:00200000:2.0:1789760732.533980:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.533985:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.534007:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22740992 i_blocks 45063 00000020:00200000:2.0:1789760732.534011:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.534019:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.534046:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22740992, count: 4096 00000080:00200000:2.0:1789760732.534052:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22740992, bytes: 4096 00000080:00200000:2.0:1789760732.534059:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.534067:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.534069:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22740992, 22745088) 0 0 00000080:00200000:2.0:1789760732.534073:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.534086:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5552, 5552] 00020000:00200000:2.0:1789760732.534093:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22740992 chunk: [22740992, 22745088) 22745088, 4096 00020000:00200000:2.0:1789760732.534097:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.534111:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5552, 5552] 00000008:00010000:2.0:1789760732.534125:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.534130:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.534138:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22740992, 22745088) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.534150:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22740992=0x15b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.534238:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.534241:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22745088 00000020:00010000:2.0:1789760732.534249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.534252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.534255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.534263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.534265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.534272:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.534278:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.534292:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760732.534389:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 22740992<>22745088 00000100:00100000:3.0:1789760732.534524:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760732.534539:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495790208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760732.594561:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972868 gen 2 00000100:00100000:3.0:1789760732.594595:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266d80 x1876699495790208/t4294972869(4294972869) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760748 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760732.594633:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760732.594639:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.594647:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.594733:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760732.594754:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495790208:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.595057:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5552, 5552] 00000080:00200000:2.0:1789760732.595070:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.595076:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.595088:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22745088 i_blocks 45063 00000020:00200000:2.0:1789760732.595092:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.595099:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.595123:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22745088, count: 4096 00000080:00200000:2.0:1789760732.595129:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22745088, bytes: 4096 00000080:00200000:2.0:1789760732.595136:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.595144:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.595146:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22745088, 22749184) 0 0 00000080:00200000:2.0:1789760732.595150:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.595162:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5553, 5553] 00020000:00200000:2.0:1789760732.595198:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22745088 chunk: [22745088, 22749184) 22749184, 4096 00020000:00200000:2.0:1789760732.595203:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.595217:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5553, 5553] 00000008:00010000:2.0:1789760732.595229:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.595235:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.595245:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22745088, 22749184) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.595256:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22745088=0x15b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.595298:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.595301:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22749184 00000020:00010000:2.0:1789760732.595307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.595310:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.595313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.595316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.595319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.595325:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.595330:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.595345:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760732.595440:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 22745088<>22749184 00000100:00100000:3.0:1789760732.595503:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495790336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760732.631472:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972870 gen 2 00000008:00010000:3.0:1789760732.631521:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760732.631529:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.631538:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.631560:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760732.631584:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495790336:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.631690:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5553, 5553] 00000080:00200000:2.0:1789760732.631703:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.631709:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.631721:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22749184 i_blocks 45063 00000020:00200000:2.0:1789760732.631724:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.631732:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.631756:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22749184, count: 4096 00000080:00200000:2.0:1789760732.631762:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22749184, bytes: 4096 00000080:00200000:2.0:1789760732.631770:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.631777:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.631779:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22749184, 22753280) 0 0 00000080:00200000:2.0:1789760732.631783:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.631796:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5554, 5554] 00020000:00200000:2.0:1789760732.631803:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22749184 chunk: [22749184, 22753280) 22753280, 4096 00020000:00200000:2.0:1789760732.631808:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.631823:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5554, 5554] 00000008:00010000:2.0:1789760732.631838:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.631843:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.631852:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22749184, 22753280) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.631863:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22749184=0x15b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.631907:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.631909:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22753280 00000020:00010000:2.0:1789760732.631916:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.631919:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.631922:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.631926:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.631929:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.631935:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.631940:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.631955:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760732.632060:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 22749184<>22753280 00000100:00100000:3.0:1789760732.632210:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495790464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760732.671236:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972871 gen 2 00000008:00010000:3.0:1789760732.671272:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760732.671278:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.671288:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.671309:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760732.671334:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495790464:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.671431:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5554, 5554] 00000080:00200000:2.0:1789760732.671445:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.671450:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.671463:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22753280 i_blocks 45063 00000020:00200000:2.0:1789760732.671466:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.671474:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.671500:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22753280, count: 4096 00000080:00200000:2.0:1789760732.671506:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22753280, bytes: 4096 00000080:00200000:2.0:1789760732.671513:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.671521:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.671523:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22753280, 22757376) 0 0 00000080:00200000:2.0:1789760732.671527:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.671540:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5555, 5555] 00020000:00200000:2.0:1789760732.671546:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22753280 chunk: [22753280, 22757376) 22757376, 4096 00020000:00200000:2.0:1789760732.671551:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.671565:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5555, 5555] 00000008:00010000:2.0:1789760732.671579:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.671584:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.671594:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22753280, 22757376) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.671604:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22753280=0x15b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.671648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.671650:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22757376 00000020:00010000:2.0:1789760732.671657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.671660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.671663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.671666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.671669:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.671683:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.671689:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.671703:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760732.671805:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 22753280<>22757376 00000100:00100000:2.0:1789760732.671851:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760732.671864:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495790592:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760732.708286:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760732.708299:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.708309:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.708331:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760732.708354:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495790592:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.770582:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5556, 5556] 00000080:00200000:2.0:1789760732.770605:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.770611:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.770626:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22761472 i_blocks 45063 00000020:00200000:2.0:1789760732.770632:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.770711:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.770739:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22761472, count: 4096 00000080:00200000:2.0:1789760732.770746:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22761472, bytes: 4096 00000080:00200000:2.0:1789760732.770757:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.770765:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.770767:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22761472, 22765568) 0 0 00000080:00200000:2.0:1789760732.770771:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.770785:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5557, 5557] 00020000:00200000:2.0:1789760732.770792:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22761472 chunk: [22761472, 22765568) 22765568, 4096 00020000:00200000:2.0:1789760732.770796:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.770813:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5557, 5557] 00000008:00010000:2.0:1789760732.770843:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.770849:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.770863:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22761472, 22765568) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.770873:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22761472=0x15b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.770927:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.770930:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22765568 00000020:00010000:2.0:1789760732.770937:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.770941:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.770943:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.770946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.770948:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.770955:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.770960:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.770983:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760732.771080:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 22761472<>22765568 00000100:00100000:3.0:1789760732.771200:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495790848:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760732.817622:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760732.817634:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.817643:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.817698:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760732.817723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495790848:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.873389:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5558, 5558] 00000080:00200000:2.0:1789760732.873401:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.873406:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.873420:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22769664 i_blocks 45063 00000020:00200000:2.0:1789760732.873424:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.873431:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.873458:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22769664, count: 4096 00000080:00200000:2.0:1789760732.873464:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22769664, bytes: 4096 00000080:00200000:2.0:1789760732.873472:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.873478:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.873480:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22769664, 22773760) 0 0 00000080:00200000:2.0:1789760732.873483:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.873495:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5559, 5559] 00020000:00200000:2.0:1789760732.873502:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22769664 chunk: [22769664, 22773760) 22773760, 4096 00020000:00200000:2.0:1789760732.873506:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.873522:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5559, 5559] 00000008:00010000:2.0:1789760732.873535:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.873540:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.873548:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22769664, 22773760) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.873558:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22769664=0x15b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.873602:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.873604:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22773760 00000020:00010000:2.0:1789760732.873611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.873614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.873616:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.873619:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.873622:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.873628:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.873633:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.873645:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760732.873811:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 22769664<>22773760 00000100:00100000:3.0:1789760732.873886:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495791104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760732.915072:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972876 gen 2 00000008:00010000:3.0:1789760732.915110:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760732.915115:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.915122:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.915136:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760732.915159:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495791104:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.915270:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5559, 5559] 00000080:00200000:2.0:1789760732.915283:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.915289:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.915300:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22773760 i_blocks 45063 00000020:00200000:2.0:1789760732.915304:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.915313:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.915337:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22773760, count: 4096 00000080:00200000:2.0:1789760732.915342:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22773760, bytes: 4096 00000080:00200000:2.0:1789760732.915349:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.915356:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.915358:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22773760, 22777856) 0 0 00000080:00200000:2.0:1789760732.915362:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.915374:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5560, 5560] 00020000:00200000:2.0:1789760732.915382:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22773760 chunk: [22773760, 22777856) 22777856, 4096 00020000:00200000:2.0:1789760732.915386:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.915403:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5560, 5560] 00000008:00010000:2.0:1789760732.915417:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.915422:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.915430:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22773760, 22777856) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.915440:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22773760=0x15b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.915485:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.915488:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22777856 00000020:00010000:2.0:1789760732.915494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.915498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.915501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.915504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.915507:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.915513:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.915519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.915532:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760732.916024:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 22773760<>22777856 00000100:00100000:3.0:1789760732.916438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495791232:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760732.983936:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972877 gen 2 00000008:00010000:3.0:1789760732.983985:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760732.984210:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.984221:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.984296:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760732.984319:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495791232:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760732.984468:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5560, 5560] 00000080:00200000:2.0:1789760732.984481:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.984486:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.984499:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22777856 i_blocks 45063 00000020:00200000:2.0:1789760732.984503:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.984511:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.984538:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22777856, count: 4096 00000080:00200000:2.0:1789760732.984546:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22777856, bytes: 4096 00000080:00200000:2.0:1789760732.984552:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.984559:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.984561:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22777856, 22781952) 0 0 00000080:00200000:2.0:1789760732.984565:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.984579:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5561, 5561] 00020000:00200000:2.0:1789760732.984587:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22777856 chunk: [22777856, 22781952) 22781952, 4096 00020000:00200000:2.0:1789760732.984591:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.984608:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5561, 5561] 00000008:00010000:2.0:1789760732.984622:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.984628:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760732.984635:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22777856, 22781952) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.984646:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22777856=0x15b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.984772:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.984775:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22781952 00000020:00010000:2.0:1789760732.984782:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.984785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.984788:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.984791:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.984794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.984801:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.984806:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.984820:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760732.985466:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 22777856<>22781952 00000100:00100000:3.0:1789760732.985526:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495791360:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760733.038774:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972878 gen 2 00000008:00010000:3.0:1789760733.038814:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760733.038820:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.038829:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.038854:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760733.038880:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495791360:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760733.070095:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2359:1876699495791616:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.070228:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2359:1876699495791872:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.070262:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760733.070269:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495791744:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.078368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495791744:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.078387:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2359:1876699495791616:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.078397:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2359:1876699495791872:192.168.203.109@tcp:400:kworker.0 00000080:00200000:3.0:1789760733.150015:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5563, 5563] 00000080:00200000:3.0:1789760733.150030:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.150035:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.150047:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22790144 i_blocks 45063 00000020:00200000:3.0:1789760733.150051:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.150060:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.150109:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22790144, count: 4096 00000080:00200000:3.0:1789760733.150117:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22790144, bytes: 4096 00000080:00200000:3.0:1789760733.150128:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.150136:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.150138:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22790144, 22794240) 0 0 00000080:00200000:3.0:1789760733.150142:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.150155:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5564, 5564] 00020000:00200000:3.0:1789760733.150163:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22790144 chunk: [22790144, 22794240) 22794240, 4096 00020000:00200000:3.0:1789760733.150194:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.150212:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5564, 5564] 00000008:00010000:3.0:1789760733.150226:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.150231:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.150239:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22790144, 22794240) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.150251:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22790144=0x15bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.150300:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.150303:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22794240 00000020:00010000:3.0:1789760733.150309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.150312:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.150315:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.150318:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.150322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.150328:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.150333:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.150348:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760733.150445:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 22790144<>22794240 00000100:00100000:3.0:1789760733.150489:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760733.150503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495792128:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760733.187469:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972881 gen 2 00000008:00010000:3.0:1789760733.187507:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760733.187514:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.187522:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.187645:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760733.187687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495792128:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.250802:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5565, 5565] 00000080:00200000:3.0:1789760733.250815:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.250820:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.250832:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22798336 i_blocks 45063 00000020:00200000:3.0:1789760733.250835:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.250843:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.250867:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22798336, count: 4096 00000080:00200000:3.0:1789760733.250873:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22798336, bytes: 4096 00000080:00200000:3.0:1789760733.250881:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.250887:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.250889:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22798336, 22802432) 0 0 00000080:00200000:3.0:1789760733.250892:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.251085:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5566, 5566] 00020000:00200000:3.0:1789760733.251093:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22798336 chunk: [22798336, 22802432) 22802432, 4096 00020000:00200000:3.0:1789760733.251097:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.251112:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5566, 5566] 00000008:00010000:3.0:1789760733.251126:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.251132:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.251141:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22798336, 22802432) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.251152:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22798336=0x15be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.251228:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.251230:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22802432 00000020:00010000:3.0:1789760733.251238:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.251241:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.251245:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.251247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.251250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.251256:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.251262:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.251276:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760733.251369:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 22798336<>22802432 00000100:00100000:2.0:1789760733.251443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495792384:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760733.308316:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972883 gen 2 00000008:00010000:2.0:1789760733.308350:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760733.308357:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.308365:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.308386:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760733.308407:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495792384:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.308527:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5566, 5566] 00000080:00200000:3.0:1789760733.308541:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.308546:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.308559:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22802432 i_blocks 45063 00000020:00200000:3.0:1789760733.308563:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.308571:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.308598:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22802432, count: 4096 00000080:00200000:3.0:1789760733.308605:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22802432, bytes: 4096 00000080:00200000:3.0:1789760733.308614:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.308621:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.308623:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22802432, 22806528) 0 0 00000080:00200000:3.0:1789760733.308626:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.308638:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5567, 5567] 00020000:00200000:3.0:1789760733.308658:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22802432 chunk: [22802432, 22806528) 22806528, 4096 00020000:00200000:3.0:1789760733.308663:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.308688:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5567, 5567] 00000008:00010000:3.0:1789760733.308702:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.309142:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.309156:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22802432, 22806528) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.309197:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22802432=0x15bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.309246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.309249:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22806528 00000020:00010000:3.0:1789760733.309255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.309258:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.309261:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.309264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.309266:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.309272:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.309277:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.309293:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760733.309387:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 22802432<>22806528 00000100:00100000:2.0:1789760733.309533:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495792512:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760733.354343:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760733.354355:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.354364:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.354384:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:2.0:1789760733.354409:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495792512:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.354435:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5567, 5567] 00000080:00200000:3.0:1789760733.354449:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.354456:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.354468:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22806528 i_blocks 45063 00000020:00200000:3.0:1789760733.354472:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.354479:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.354503:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22806528, count: 4096 00000080:00200000:3.0:1789760733.354509:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22806528, bytes: 4096 00000080:00200000:3.0:1789760733.354516:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.354523:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.354526:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22806528, 22810624) 0 0 00000080:00200000:3.0:1789760733.354529:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.354542:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5568, 5568] 00020000:00200000:3.0:1789760733.354550:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22806528 chunk: [22806528, 22810624) 22810624, 4096 00020000:00200000:3.0:1789760733.354554:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.354568:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5568, 5568] 00000008:00010000:3.0:1789760733.354582:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.354589:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.354597:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22806528, 22810624) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.354608:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22806528=0x15c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.354655:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.354658:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22810624 00000020:00010000:3.0:1789760733.354665:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.354682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.354685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.354687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.354690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.354697:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.354703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.355209:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760733.355307:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 22806528<>22810624 00000100:00100000:2.0:1789760733.355581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495792640:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760733.398459:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972885 gen 2 00000008:00010000:2.0:1789760733.398504:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760733.398511:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.398520:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.398541:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760733.398564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495792640:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.398568:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5568, 5568] 00000080:00200000:3.0:1789760733.398581:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.398586:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.398597:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22810624 i_blocks 45063 00000020:00200000:3.0:1789760733.398600:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.398607:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.398631:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22810624, count: 4096 00000080:00200000:3.0:1789760733.398637:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22810624, bytes: 4096 00000080:00200000:3.0:1789760733.398643:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.398650:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.398652:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22810624, 22814720) 0 0 00000080:00200000:3.0:1789760733.398655:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.398676:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5569, 5569] 00020000:00200000:3.0:1789760733.398749:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22810624 chunk: [22810624, 22814720) 22814720, 4096 00020000:00200000:3.0:1789760733.398753:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.398768:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5569, 5569] 00000008:00010000:3.0:1789760733.398782:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.398788:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.398796:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22810624, 22814720) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.398807:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22810624=0x15c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.398846:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.398849:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22814720 00000020:00010000:3.0:1789760733.398855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.398859:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.398861:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.398865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.398868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.398873:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.398877:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.398889:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760733.398977:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 22810624<>22814720 00000100:00100000:2.0:1789760733.399041:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495792768:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760733.454514:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972886 gen 2 00000008:00010000:2.0:1789760733.454550:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760733.454556:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.454564:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.454622:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760733.454644:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495792768:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.454731:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5569, 5569] 00000080:00200000:3.0:1789760733.454745:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.454750:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.454761:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22814720 i_blocks 45063 00000020:00200000:3.0:1789760733.454764:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.454772:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.454795:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22814720, count: 4096 00000080:00200000:3.0:1789760733.454801:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22814720, bytes: 4096 00000080:00200000:3.0:1789760733.454808:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.454814:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.454816:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22814720, 22818816) 0 0 00000080:00200000:3.0:1789760733.454820:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.454832:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5570, 5570] 00020000:00200000:3.0:1789760733.454839:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22814720 chunk: [22814720, 22818816) 22818816, 4096 00020000:00200000:3.0:1789760733.454843:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.454857:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5570, 5570] 00000008:00010000:3.0:1789760733.454870:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.454876:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.454884:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22814720, 22818816) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.454894:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22814720=0x15c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.455233:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.455236:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22818816 00000020:00010000:3.0:1789760733.455244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.455247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.455250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.455253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.455256:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.455263:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.455268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.455282:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760733.455460:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 22814720<>22818816 00000100:00100000:2.0:1789760733.455559:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495792896:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760733.504544:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760733.504555:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.504564:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.504582:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760733.504605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495792896:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.504745:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5570, 5570] 00000080:00200000:3.0:1789760733.504762:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.504768:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.504788:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22818816 i_blocks 45063 00000020:00200000:3.0:1789760733.504792:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.504807:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.504832:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22818816, count: 4096 00000080:00200000:3.0:1789760733.504838:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22818816, bytes: 4096 00000080:00200000:3.0:1789760733.504846:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.504853:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.504855:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22818816, 22822912) 0 0 00000080:00200000:3.0:1789760733.504869:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.504881:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5571, 5571] 00020000:00200000:3.0:1789760733.504888:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22818816 chunk: [22818816, 22822912) 22822912, 4096 00020000:00200000:3.0:1789760733.504892:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.504907:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5571, 5571] 00000008:00010000:3.0:1789760733.504919:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.504925:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.504934:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22818816, 22822912) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.504945:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22818816=0x15c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.504990:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.504993:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22822912 00000020:00010000:3.0:1789760733.504999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.505003:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.505006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.505009:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.505012:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.505305:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.505313:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.505330:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760733.505425:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 22818816<>22822912 00000100:00100000:2.0:1789760733.505492:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495793024:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760733.553795:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972888 gen 2 00000008:00010000:2.0:1789760733.553845:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760733.553853:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.553862:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.553887:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760733.553912:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495793024:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.554547:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5571, 5571] 00000080:00200000:3.0:1789760733.554561:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.554567:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.554579:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22822912 i_blocks 45063 00000020:00200000:3.0:1789760733.554583:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.554590:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.554613:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22822912, count: 4096 00000080:00200000:3.0:1789760733.554619:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22822912, bytes: 4096 00000080:00200000:3.0:1789760733.554629:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.554636:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.554638:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22822912, 22827008) 0 0 00000080:00200000:3.0:1789760733.554642:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.554653:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5572, 5572] 00020000:00200000:3.0:1789760733.554660:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22822912 chunk: [22822912, 22827008) 22827008, 4096 00020000:00200000:3.0:1789760733.554664:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.554692:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5572, 5572] 00000008:00010000:3.0:1789760733.554706:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.554712:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.554720:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22822912, 22827008) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.554730:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22822912=0x15c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.554772:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.554775:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22827008 00000020:00010000:3.0:1789760733.554781:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.554784:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.554787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.554790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.554792:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.554811:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.554817:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.554831:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760733.554928:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 22822912<>22827008 00000100:00100000:2.0:1789760733.555077:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495793152:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760733.610739:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972889 gen 2 00000008:00010000:2.0:1789760733.610776:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760733.610783:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.610792:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.610812:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760733.610835:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495793152:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.611000:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5572, 5572] 00000080:00200000:3.0:1789760733.611012:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.611017:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.611030:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22827008 i_blocks 45063 00000020:00200000:3.0:1789760733.611033:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.611042:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.611066:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22827008, count: 4096 00000080:00200000:3.0:1789760733.611072:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22827008, bytes: 4096 00000080:00200000:3.0:1789760733.611080:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.611087:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.611089:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22827008, 22831104) 0 0 00000080:00200000:3.0:1789760733.611092:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.611104:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5573, 5573] 00020000:00200000:3.0:1789760733.611112:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22827008 chunk: [22827008, 22831104) 22831104, 4096 00020000:00200000:3.0:1789760733.611115:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.611129:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5573, 5573] 00000008:00010000:3.0:1789760733.611142:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.611147:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.611155:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22827008, 22831104) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.611165:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22827008=0x15c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.611234:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.611236:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22831104 00000020:00010000:3.0:1789760733.611244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.611247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.611250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.611253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.611256:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.611262:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.611267:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.611281:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760733.611373:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 22827008<>22831104 00000100:00100000:2.0:1789760733.611511:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760733.611526:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495793280:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760733.645684:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760733.670028:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972890 gen 2 00000008:00010000:2.0:1789760733.670067:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760733.670074:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.670084:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.670106:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:2.0:1789760733.670131:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495793280:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760733.670266:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5573, 5573] 00000080:00200000:3.0:1789760733.670279:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760733.670284:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760733.670297:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22831104 i_blocks 45063 00000020:00200000:3.0:1789760733.670300:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760733.670308:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760733.670332:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22831104, count: 4096 00000080:00200000:3.0:1789760733.670338:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22831104, bytes: 4096 00000080:00200000:3.0:1789760733.670345:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760733.670352:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760733.670354:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22831104, 22835200) 0 0 00000080:00200000:3.0:1789760733.670359:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760733.670370:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5574, 5574] 00020000:00200000:3.0:1789760733.670377:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22831104 chunk: [22831104, 22835200) 22835200, 4096 00020000:00200000:3.0:1789760733.670381:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760733.670395:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5574, 5574] 00000008:00010000:3.0:1789760733.670408:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760733.670427:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760733.670436:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22831104, 22835200) append 0 append lockpos 0 00000080:00200000:3.0:1789760733.670446:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22831104=0x15c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760733.670486:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760733.670489:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22835200 00000020:00010000:3.0:1789760733.670496:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760733.670499:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.670501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760733.670504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760733.670507:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760733.670513:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760733.670518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760733.670531:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760733.670620:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 22831104<>22835200 00000100:00100000:3.0:1789760733.670654:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760733.670680:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495793408:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760733.764203:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:3.0:1789760733.764214:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.764223:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.764245:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:3.0:1789760733.764270:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495793408:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760734.606805:0:468:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1789760734.606822:0:468:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.606880:0:468:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760734.666418:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5589, 5589] 00000080:00200000:3.0:1789760734.666431:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760734.666436:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760734.666448:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22896640 i_blocks 45063 00000020:00200000:3.0:1789760734.666451:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760734.666459:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760734.666484:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22896640, count: 4096 00000080:00200000:3.0:1789760734.666491:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22896640, bytes: 4096 00000080:00200000:3.0:1789760734.666499:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760734.666507:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760734.666508:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22896640, 22900736) 0 0 00000080:00200000:3.0:1789760734.666513:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760734.666525:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5590, 5590] 00020000:00200000:3.0:1789760734.666532:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22896640 chunk: [22896640, 22900736) 22900736, 4096 00020000:00200000:3.0:1789760734.666535:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760734.666551:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5590, 5590] 00000008:00010000:3.0:1789760734.666565:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760734.666570:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760734.666579:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22896640, 22900736) append 0 append lockpos 0 00000080:00200000:3.0:1789760734.666590:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22896640=0x15d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760734.666634:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760734.666636:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22900736 00000020:00010000:3.0:1789760734.666642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760734.666645:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760734.666647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760734.666650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760734.666652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760734.666670:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760734.666679:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760734.666694:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760734.666796:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 22896640<>22900736 00000100:00100000:2.0:1789760734.666958:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760734.666985:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495795456:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760734.744393:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972907 gen 2 00000008:00010000:3.0:1789760734.744430:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:3.0:1789760734.744436:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.744455:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.744472:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:3.0:1789760734.744495:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495795456:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760734.805041:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5591, 5591] 00000080:00200000:3.0:1789760734.805056:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760734.805061:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760734.805074:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22904832 i_blocks 45063 00000020:00200000:3.0:1789760734.805077:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760734.805086:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760734.805111:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22904832, count: 4096 00000080:00200000:3.0:1789760734.805118:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22904832, bytes: 4096 00000080:00200000:3.0:1789760734.805127:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760734.805134:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760734.805135:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22904832, 22908928) 0 0 00000080:00200000:3.0:1789760734.805139:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760734.805152:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5592, 5592] 00020000:00200000:3.0:1789760734.805160:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22904832 chunk: [22904832, 22908928) 22908928, 4096 00020000:00200000:3.0:1789760734.805164:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760734.805213:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5592, 5592] 00000008:00010000:3.0:1789760734.805227:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760734.805232:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760734.805240:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22904832, 22908928) append 0 append lockpos 0 00000080:00200000:3.0:1789760734.805251:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22904832=0x15d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760734.805289:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760734.805292:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22908928 00000020:00010000:3.0:1789760734.805299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760734.805302:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760734.805305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760734.805308:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760734.805311:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760734.805317:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760734.805323:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760734.851954:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5592, 5592] 00000080:00200000:3.0:1789760734.851971:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760734.851977:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.850804:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5608, 5608] 00000080:00200000:2.0:1789760735.850816:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.850821:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.850833:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22974464 i_blocks 45063 00000020:00200000:2.0:1789760735.850837:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.850845:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.850879:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22974464, count: 4096 00000080:00200000:2.0:1789760735.850885:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22974464, bytes: 4096 00000080:00200000:2.0:1789760735.850893:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.850900:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.850902:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22974464, 22978560) 0 0 00000080:00200000:2.0:1789760735.850905:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.850918:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5609, 5609] 00020000:00200000:2.0:1789760735.850926:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22974464 chunk: [22974464, 22978560) 22978560, 4096 00020000:00200000:2.0:1789760735.850952:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.850967:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5609, 5609] 00000008:00010000:2.0:1789760735.850993:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.850999:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760735.851009:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22974464, 22978560) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.851020:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=22974464=0x15e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.851064:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.851066:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22978560 00000020:00010000:2.0:1789760735.851073:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.851076:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.851079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.851082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.851085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.851091:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.851097:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.851110:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760735.851243:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 22974464<>22978560 00000100:00100000:2.0:1789760735.851281:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760735.851294:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495797888:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760735.893836:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972926 gen 2 00000008:00010000:2.0:1789760735.894567:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760735.894576:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760735.894587:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760735.894612:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760735.894666:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495797888:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760736.054414:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760736.054423:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.054433:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.054453:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760736.054483:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495798400:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.055793:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495798528:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.097527:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972931 gen 2 00000008:00010000:1.0:1789760736.097579:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760736.097587:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.097599:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.097898:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760736.097935:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495798528:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.099196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495798656:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.146278:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972932 gen 2 00000008:00010000:1.0:1789760736.146330:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760736.146338:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.146348:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.146370:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760736.146400:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495798656:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.148505:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760736.148609:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495798784:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.198835:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972933 gen 2 00000008:00010000:1.0:1789760736.199003:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760736.199013:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.199024:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.199153:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760736.199229:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495798784:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760736.199419:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5616, 5616] 00000080:00200000:2.0:1789760736.199437:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.199443:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.199460:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23007232 i_blocks 45063 00000020:00200000:2.0:1789760736.199464:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.199474:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.199509:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23007232, count: 4096 00000080:00200000:2.0:1789760736.199517:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23007232, bytes: 4096 00000080:00200000:2.0:1789760736.199527:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.199537:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.199540:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23007232, 23011328) 0 0 00000080:00200000:2.0:1789760736.199544:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.199561:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5617, 5617] 00020000:00200000:2.0:1789760736.199569:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23007232 chunk: [23007232, 23011328) 23011328, 4096 00020000:00200000:2.0:1789760736.199574:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.199601:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5617, 5617] 00000008:00010000:2.0:1789760736.199619:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.199698:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760736.199712:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23007232, 23011328) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.199728:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23007232=0x15f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.199783:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.199786:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23011328 00000020:00010000:2.0:1789760736.199795:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.199799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.199802:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.199806:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce128 (ffff8dd218376ab8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.199809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376ab8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.199816:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.199823:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.199842:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760736.200277:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 23007232<>23011328 00000100:00100000:3.0:1789760736.200375:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495798912:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760736.244717:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972934 gen 2 00000008:00010000:3.0:1789760736.244769:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760736.244778:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.244791:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.244854:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760736.244889:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495798912:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760736.295122:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760736.295138:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.295150:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.295200:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760736.295228:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495799040:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.296726:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495799168:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.333072:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972936 gen 2 00000008:00010000:1.0:1789760736.333142:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760736.333150:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.333161:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.333264:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760736.333293:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495799168:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.334206:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495799296:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.379763:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972937 gen 2 00000008:00010000:1.0:1789760736.379812:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760736.379821:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.379833:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.379859:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760736.379893:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495799296:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760736.379898:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5620, 5620] 00000080:00200000:3.0:1789760736.379920:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760736.379927:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760736.379944:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23023616 i_blocks 45063 00000020:00200000:3.0:1789760736.379948:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760736.379959:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760736.380038:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23023616, count: 4096 00000080:00200000:3.0:1789760736.380047:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23023616, bytes: 4096 00000080:00200000:3.0:1789760736.380063:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760736.380072:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760736.380074:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23023616, 23027712) 0 0 00000080:00200000:3.0:1789760736.380079:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760736.380095:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5621, 5621] 00020000:00200000:3.0:1789760736.380104:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23023616 chunk: [23023616, 23027712) 23027712, 4096 00020000:00200000:3.0:1789760736.380109:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760736.380129:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5621, 5621] 00000008:00010000:3.0:1789760736.380148:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760736.380154:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760736.380199:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23023616, 23027712) append 0 append lockpos 0 00000080:00200000:3.0:1789760736.380215:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23023616=0x15f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760736.380273:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.380276:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23027712 00000020:00010000:3.0:1789760736.380285:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760736.380289:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.380293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.380296:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760736.380299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760736.380307:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760736.380314:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760736.380330:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760736.380666:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 23023616<>23027712 00000100:00100000:2.0:1789760736.380745:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495799424:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760736.414794:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972938 gen 2 00000008:00010000:2.0:1789760736.414845:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760736.414854:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760736.414865:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760736.416091:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760736.416139:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495799424:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760736.420044:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5621, 5621] 00000080:00200000:1.0:1789760736.420079:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760736.420088:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000008:00010000:1.0:1789760736.464715:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760736.464729:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.464740:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.464761:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760736.464790:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495799552:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760736.465115:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5622, 5622] 00000080:00200000:2.0:1789760736.465133:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.465140:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.465156:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23031808 i_blocks 45063 00000020:00200000:2.0:1789760736.465159:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.465209:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.465243:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23031808, count: 4096 00000080:00200000:2.0:1789760736.465250:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23031808, bytes: 4096 00000080:00200000:2.0:1789760736.465261:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.465269:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.465272:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23031808, 23035904) 0 0 00000080:00200000:2.0:1789760736.465277:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.465293:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5623, 5623] 00020000:00200000:2.0:1789760736.465302:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23031808 chunk: [23031808, 23035904) 23035904, 4096 00020000:00200000:2.0:1789760736.465306:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.465606:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5623, 5623] 00000008:00010000:2.0:1789760736.465627:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.465632:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760736.465643:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23031808, 23035904) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.465666:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23031808=0x15f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.465720:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.465723:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23035904 00000020:00010000:2.0:1789760736.465731:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.465734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.465737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.465742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.465744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.465752:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.465759:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.465777:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760736.465902:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 23031808<>23035904 00000100:00100000:2.0:1789760736.465959:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760736.467817:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495799680:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760736.517381:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972940 gen 2 00000008:00010000:2.0:1789760736.517447:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:2.0:1789760736.517457:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760736.517467:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760736.517495:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:2.0:1789760736.518283:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495799680:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.519726:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760736.519751:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495799808:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760736.572271:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760736.572287:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.572299:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.572319:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760736.572349:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495799808:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.573165:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495799936:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.626719:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972942 gen 2 00000008:00010000:1.0:1789760736.626798:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760736.626808:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.626818:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.626844:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760736.626881:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495799936:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760736.629160:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760736.629223:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495800064:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760736.675484:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760736.675501:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.675514:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.675536:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760736.675569:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495800064:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760736.717474:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760736.717795:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972943 gen 2 00000100:00100000:1.0:1789760736.717832:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211834700 x1876699495800192/t4294972944(4294972944) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760752 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760736.717880:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760736.717887:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.717897:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.717921:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760736.717945:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495800192:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760736.718052:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5627, 5627] 00000080:00200000:2.0:1789760736.718069:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.718075:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.718091:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23052288 i_blocks 45063 00000020:00200000:2.0:1789760736.718094:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.718104:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.718138:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23052288, count: 4096 00000080:00200000:2.0:1789760736.718144:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23052288, bytes: 4096 00000080:00200000:2.0:1789760736.718154:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.718163:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.718166:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23052288, 23056384) 0 0 00000080:00200000:2.0:1789760736.718616:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.718636:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5628, 5628] 00020000:00200000:2.0:1789760736.718654:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23052288 chunk: [23052288, 23056384) 23056384, 4096 00020000:00200000:2.0:1789760736.718660:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.718682:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5628, 5628] 00000008:00010000:2.0:1789760736.718762:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.718769:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760736.718782:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23052288, 23056384) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.718797:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23052288=0x15fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.718860:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.718864:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23056384 00000020:00010000:2.0:1789760736.718874:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.718877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.718881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.718884:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.718887:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.718895:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.718902:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.718921:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760736.719049:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 23052288<>23056384 00000100:00100000:3.0:1789760736.719278:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760736.719311:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495800320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760736.770536:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972945 gen 2 00000008:00010000:3.0:1789760736.770608:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760736.770617:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.770629:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.770665:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760736.770695:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495800320:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760736.770808:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5628, 5628] 00000080:00200000:2.0:1789760736.770897:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.770904:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.770921:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23056384 i_blocks 45063 00000020:00200000:2.0:1789760736.770925:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.770935:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.771037:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23056384, count: 4096 00000080:00200000:2.0:1789760736.771146:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23056384, bytes: 4096 00000080:00200000:2.0:1789760736.771158:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.771192:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.771195:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23056384, 23060480) 0 0 00000080:00200000:2.0:1789760736.771199:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.771217:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5629, 5629] 00020000:00200000:2.0:1789760736.771226:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23056384 chunk: [23056384, 23060480) 23060480, 4096 00020000:00200000:2.0:1789760736.771231:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.771253:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5629, 5629] 00000008:00010000:2.0:1789760736.771284:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.771291:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760736.771304:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23056384, 23060480) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.771319:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23056384=0x15fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.771627:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.771631:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23060480 00000020:00010000:2.0:1789760736.771652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.771656:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.771660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.771664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.771668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.771676:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.771683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.771703:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760736.773721:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23056384<>23060480 00000100:00100000:2.0:1789760736.773835:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760736.773865:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495800448:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760736.820605:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760736.820620:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760736.820631:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760736.820661:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000080:00200000:1.0:1789760736.821274:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5629, 5629] 00000080:00200000:1.0:1789760736.821294:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760736.821301:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760736.821316:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23060480 i_blocks 45063 00000020:00200000:1.0:1789760736.821321:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760736.821330:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760736.821378:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23060480, count: 4096 00000080:00200000:1.0:1789760736.821387:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23060480, bytes: 4096 00000080:00200000:1.0:1789760736.821402:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760736.821412:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760736.821415:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23060480, 23064576) 0 0 00000080:00200000:1.0:1789760736.821420:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760736.821443:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5630, 5630] 00020000:00200000:1.0:1789760736.821453:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23060480 chunk: [23060480, 23064576) 23064576, 4096 00020000:00200000:1.0:1789760736.821458:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760736.821480:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5630, 5630] 00000008:00010000:1.0:1789760736.821505:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760736.821512:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760736.821523:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23060480, 23064576) append 0 append lockpos 0 00000080:00200000:1.0:1789760736.821539:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23060480=0x15fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760736.821592:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760736.821595:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23064576 00000020:00010000:1.0:1789760736.821603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760736.821607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760736.821611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760736.821615:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760736.821618:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760736.821625:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760736.821634:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1789760736.821957:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495800448:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760736.822014:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:2.0:1789760736.822149:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 23060480<>23064576 00000100:00100000:3.0:1789760736.822374:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495800576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760736.879757:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972947 gen 2 00000008:00010000:3.0:1789760736.879821:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:3.0:1789760736.879829:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.879904:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.880034:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:3.0:1789760736.880069:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495800576:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760736.880154:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5630, 5630] 00000080:00200000:1.0:1789760736.880259:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760736.880267:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760736.880286:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23064576 i_blocks 45063 00000020:00200000:1.0:1789760736.880289:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760736.880300:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760736.880334:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23064576, count: 4096 00000080:00200000:1.0:1789760736.880341:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23064576, bytes: 4096 00000080:00200000:1.0:1789760736.880353:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760736.880361:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760736.880363:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23064576, 23068672) 0 0 00000080:00200000:1.0:1789760736.880367:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760736.880447:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5631, 5631] 00020000:00200000:1.0:1789760736.880457:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23064576 chunk: [23064576, 23068672) 23068672, 4096 00020000:00200000:1.0:1789760736.880463:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760736.880486:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5631, 5631] 00000008:00010000:1.0:1789760736.880507:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760736.880514:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760736.880527:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23064576, 23068672) append 0 append lockpos 0 00000080:00200000:1.0:1789760736.880695:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23064576=0x15ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760736.880760:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760736.880763:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23068672 00000020:00010000:1.0:1789760736.880773:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760736.880777:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760736.880781:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760736.880785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760736.880789:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760736.880799:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760736.880806:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760736.880824:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760736.881075:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 23064576<>23068672 00000100:00100000:1.0:1789760736.918790:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972948 gen 2 00000008:00010000:1.0:1789760736.919129:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760736.919138:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.919148:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.919245:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760736.919268:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495800704:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760736.919834:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5631, 5631] 00000080:00200000:2.0:1789760736.919853:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.919859:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.919876:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23068672 i_blocks 45063 00000020:00200000:2.0:1789760736.920320:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.920334:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.920370:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23068672, count: 4096 00000080:00200000:2.0:1789760736.920378:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23068672, bytes: 4096 00000080:00200000:2.0:1789760736.920392:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.920402:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.920404:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23068672, 23072768) 0 0 00000080:00200000:2.0:1789760736.920409:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.920428:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5632, 5632] 00020000:00200000:2.0:1789760736.920436:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23068672 chunk: [23068672, 23072768) 23072768, 4096 00020000:00200000:2.0:1789760736.920441:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.920841:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5632, 5632] 00000008:00010000:2.0:1789760736.920864:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.920870:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760736.920883:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23068672, 23072768) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.920898:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23068672=0x1600000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.921275:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.921278:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23072768 00000020:00010000:2.0:1789760736.921288:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.921291:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.921294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.921298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.921301:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.921364:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.921373:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.921399:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760736.921750:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 23068672<>23072768 00000100:00100000:3.0:1789760736.921843:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495800832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760736.957884:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972949 gen 2 00000008:00010000:3.0:1789760736.957933:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760736.957942:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.957954:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.958029:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760736.958069:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495800832:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760736.959927:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5632, 5632] 00000080:00200000:2.0:1789760736.959957:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.959965:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.959985:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23072768 i_blocks 45063 00000020:00200000:2.0:1789760736.959991:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.960122:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.960269:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23072768, count: 4096 00000080:00200000:2.0:1789760736.960278:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23072768, bytes: 4096 00000080:00200000:2.0:1789760736.960294:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.960303:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.960306:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23072768, 23076864) 0 0 00000080:00200000:2.0:1789760736.960311:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.960328:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5633, 5633] 00020000:00200000:2.0:1789760736.960336:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23072768 chunk: [23072768, 23076864) 23076864, 4096 00020000:00200000:2.0:1789760736.960341:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.960363:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5633, 5633] 00000008:00010000:2.0:1789760736.960382:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.960388:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760736.960400:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23072768, 23076864) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.960414:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23072768=0x1601000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.960482:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.960485:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23076864 00000020:00010000:2.0:1789760736.960494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.960498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.960500:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.960504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.960507:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.960515:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.960522:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.960678:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760736.960813:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23072768<>23076864 00000100:00100000:3.0:1789760736.960911:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495800960:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760737.020132:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972950 gen 2 00000008:00010000:3.0:1789760737.020360:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760737.020370:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.020380:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.020438:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760737.020468:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495800960:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.021500:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495801088:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.061440:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972951 gen 2 00000008:00010000:1.0:1789760737.061493:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760737.061501:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.061512:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.061533:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760737.061566:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495801088:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760737.127487:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760737.127505:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.127518:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.127546:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760737.127576:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495801216:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.127822:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5635, 5635] 00000080:00200000:2.0:1789760737.127838:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.127844:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.127870:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23085056 i_blocks 47111 00000020:00200000:2.0:1789760737.127875:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.127885:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.127916:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23085056, count: 4096 00000080:00200000:2.0:1789760737.127923:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23085056, bytes: 4096 00000080:00200000:2.0:1789760737.127933:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.127941:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.127943:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23085056, 23089152) 0 0 00000080:00200000:2.0:1789760737.127947:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.127964:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5636, 5636] 00020000:00200000:2.0:1789760737.127971:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23085056 chunk: [23085056, 23089152) 23089152, 4096 00020000:00200000:2.0:1789760737.127976:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.127995:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5636, 5636] 00000008:00010000:2.0:1789760737.128012:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.128017:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.128027:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23085056, 23089152) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.128039:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23085056=0x1604000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.128281:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.128285:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23089152 00000020:00010000:2.0:1789760737.128294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.128298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.128301:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.128305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.128307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.128315:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.128322:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.128334:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760737.128464:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 23085056<>23089152 00000100:00100000:3.0:1789760737.128546:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801344:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760737.167916:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972953 gen 2 00000008:00010000:3.0:1789760737.167995:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760737.168004:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.168015:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.168042:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760737.168071:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801344:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.168221:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5636, 5636] 00000080:00200000:2.0:1789760737.168237:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.168243:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.168259:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23089152 i_blocks 47111 00000020:00200000:2.0:1789760737.168263:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.168273:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.168306:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23089152, count: 4096 00000080:00200000:2.0:1789760737.168380:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23089152, bytes: 4096 00000080:00200000:2.0:1789760737.168392:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.168401:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.168404:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23089152, 23093248) 0 0 00000080:00200000:2.0:1789760737.168408:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.168425:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5637, 5637] 00020000:00200000:2.0:1789760737.168433:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23089152 chunk: [23089152, 23093248) 23093248, 4096 00020000:00200000:2.0:1789760737.168437:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.168458:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5637, 5637] 00000008:00010000:2.0:1789760737.168475:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.168482:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.168494:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23089152, 23093248) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.168508:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23089152=0x1605000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.168575:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.168578:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23093248 00000020:00010000:2.0:1789760737.168587:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.168591:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.168594:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.168598:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.168601:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.168609:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.168722:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.168743:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760737.168867:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 23089152<>23093248 00000100:00100000:3.0:1789760737.169477:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760737.169500:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801472:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760737.209927:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760737.209942:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.209954:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.210076:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760737.210107:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801472:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.211740:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5637, 5637] 00000080:00200000:2.0:1789760737.211765:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.211772:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.211791:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23093248 i_blocks 47111 00000020:00200000:2.0:1789760737.211797:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.211809:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.211844:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23093248, count: 4096 00000080:00200000:2.0:1789760737.211852:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23093248, bytes: 4096 00000080:00200000:2.0:1789760737.211867:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.211876:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.211879:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23093248, 23097344) 0 0 00000080:00200000:2.0:1789760737.211884:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.211902:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5638, 5638] 00020000:00200000:2.0:1789760737.211910:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23093248 chunk: [23093248, 23097344) 23097344, 4096 00020000:00200000:2.0:1789760737.211915:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.211936:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5638, 5638] 00000008:00010000:2.0:1789760737.211956:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.211963:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.211975:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23093248, 23097344) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.211990:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23093248=0x1606000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.212066:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.212070:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23097344 00000020:00010000:2.0:1789760737.212079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.212082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.212085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.212089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.212092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.212100:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.212107:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.212128:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760737.212319:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 23093248<>23097344 00000100:00100000:3.0:1789760737.213154:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801600:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760737.262415:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972954 gen 2 00000100:00100000:3.0:1789760737.262457:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266680 x1876699495801600/t4294972955(4294972955) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760753 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760737.262510:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760737.262517:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.262526:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.262553:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760737.262578:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801600:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.262699:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5638, 5638] 00000080:00200000:2.0:1789760737.262783:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.262790:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.262807:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23097344 i_blocks 47111 00000020:00200000:2.0:1789760737.262811:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.262822:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.262857:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23097344, count: 4096 00000080:00200000:2.0:1789760737.262864:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23097344, bytes: 4096 00000080:00200000:2.0:1789760737.262873:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.262882:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.262884:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23097344, 23101440) 0 0 00000080:00200000:2.0:1789760737.262888:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.262904:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5639, 5639] 00020000:00200000:2.0:1789760737.267284:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23097344 chunk: [23097344, 23101440) 23101440, 4096 00020000:00200000:2.0:1789760737.267300:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.267335:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5639, 5639] 00000008:00010000:2.0:1789760737.267361:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.267367:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.267383:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23097344, 23101440) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.267401:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23097344=0x1607000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.267478:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.267482:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23101440 00000020:00010000:2.0:1789760737.267490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.267494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.267498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.267501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.267504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.267516:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.267525:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.267555:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760737.267825:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 23097344<>23101440 00000100:00100000:3.0:1789760737.269872:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760737.269911:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801728:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760737.327116:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972956 gen 2 00000008:00010000:3.0:1789760737.327207:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760737.327215:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.327226:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.327249:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760737.327278:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801728:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.328784:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5639, 5639] 00000080:00200000:2.0:1789760737.328809:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.328817:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.328835:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23101440 i_blocks 47111 00000020:00200000:2.0:1789760737.328841:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.328855:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.328889:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23101440, count: 4096 00000080:00200000:2.0:1789760737.328897:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23101440, bytes: 4096 00000080:00200000:2.0:1789760737.328910:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.328920:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.328922:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23101440, 23105536) 0 0 00000080:00200000:2.0:1789760737.328926:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.328945:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5640, 5640] 00020000:00200000:2.0:1789760737.328953:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23101440 chunk: [23101440, 23105536) 23105536, 4096 00020000:00200000:2.0:1789760737.328958:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.328978:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5640, 5640] 00000008:00010000:2.0:1789760737.328996:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.329001:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.329013:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23101440, 23105536) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.329026:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23101440=0x1608000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.329091:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.329095:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23105536 00000020:00010000:2.0:1789760737.329103:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.329106:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.329109:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.329112:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.329116:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.329123:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.329131:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.329154:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760737.329302:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 23101440<>23105536 00000100:00100000:3.0:1789760737.329383:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801856:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760737.367832:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972957 gen 2 00000008:00010000:3.0:1789760737.367897:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760737.367906:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.367917:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.367944:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760737.367984:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495801856:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.368262:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5640, 5640] 00000080:00200000:2.0:1789760737.368282:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.368289:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.368306:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23105536 i_blocks 47111 00000020:00200000:2.0:1789760737.368311:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.368321:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.368357:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23105536, count: 4096 00000080:00200000:2.0:1789760737.368364:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23105536, bytes: 4096 00000080:00200000:2.0:1789760737.368451:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.368462:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.368464:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23105536, 23109632) 0 0 00000080:00200000:2.0:1789760737.368469:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.368486:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5641, 5641] 00020000:00200000:2.0:1789760737.368495:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23105536 chunk: [23105536, 23109632) 23109632, 4096 00020000:00200000:2.0:1789760737.368499:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.368521:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5641, 5641] 00000008:00010000:2.0:1789760737.368539:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.368544:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.368556:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23105536, 23109632) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.368572:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23105536=0x1609000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.368759:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.368762:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23109632 00000020:00010000:2.0:1789760737.368771:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.368775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.368778:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.368782:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.368785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.368792:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.368800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.368817:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760737.368959:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 23105536<>23109632 00000100:00100000:3.0:1789760737.369322:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495801984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760737.420126:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972958 gen 2 00000008:00010000:3.0:1789760737.420228:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760737.420238:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.420250:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.420275:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760737.420306:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495801984:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.420881:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5641, 5641] 00000080:00200000:2.0:1789760737.420898:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.420907:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.420934:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23109632 i_blocks 47111 00000020:00200000:2.0:1789760737.420939:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.420950:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.421359:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23109632, count: 4096 00000080:00200000:2.0:1789760737.421468:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23109632, bytes: 4096 00000080:00200000:2.0:1789760737.421483:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.421491:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.421494:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23109632, 23113728) 0 0 00000080:00200000:2.0:1789760737.421498:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.421514:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5642, 5642] 00020000:00200000:2.0:1789760737.421523:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23109632 chunk: [23109632, 23113728) 23113728, 4096 00020000:00200000:2.0:1789760737.421527:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.421548:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5642, 5642] 00000008:00010000:2.0:1789760737.421565:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.421571:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.421581:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23109632, 23113728) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.421596:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23109632=0x160a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.421666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.421669:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23113728 00000020:00010000:2.0:1789760737.421677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.421680:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.421684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.421687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.421690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.421699:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.421706:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.421726:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760737.421911:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 23109632<>23113728 00000100:00100000:3.0:1789760737.422106:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495802112:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760737.472037:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972959 gen 2 00000008:00010000:3.0:1789760737.472085:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760737.472093:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.472103:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.472128:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760737.472207:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495802112:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.472604:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5642, 5642] 00000080:00200000:2.0:1789760737.472633:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.472647:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.472664:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23113728 i_blocks 47111 00000020:00200000:2.0:1789760737.472669:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.472679:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.472714:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23113728, count: 4096 00000080:00200000:2.0:1789760737.472722:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23113728, bytes: 4096 00000080:00200000:2.0:1789760737.472734:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.472742:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.472745:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23113728, 23117824) 0 0 00000080:00200000:2.0:1789760737.472749:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.472767:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5643, 5643] 00020000:00200000:2.0:1789760737.472775:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23113728 chunk: [23113728, 23117824) 23117824, 4096 00020000:00200000:2.0:1789760737.472780:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.472801:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5643, 5643] 00000008:00010000:2.0:1789760737.472819:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.472824:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.472838:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23113728, 23117824) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.472854:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23113728=0x160b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.472919:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.472922:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23117824 00000020:00010000:2.0:1789760737.472931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.472934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.472937:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.472941:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.472944:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.472951:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.472958:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.472976:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760737.473117:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 23113728<>23117824 00000100:00100000:3.0:1789760737.473811:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495802240:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760737.511500:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760737.511517:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760737.511528:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760737.511555:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760737.511588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495802240:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760737.511630:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5643, 5643] 00000080:00200000:1.0:1789760737.511681:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760737.511688:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760737.511711:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23117824 i_blocks 47111 00000020:00200000:1.0:1789760737.511715:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760737.511726:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760737.511759:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23117824, count: 4096 00000080:00200000:1.0:1789760737.511766:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23117824, bytes: 4096 00000080:00200000:1.0:1789760737.511776:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760737.511785:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760737.511787:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23117824, 23121920) 0 0 00000080:00200000:1.0:1789760737.511792:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760737.511806:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5644, 5644] 00020000:00200000:1.0:1789760737.511814:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23117824 chunk: [23117824, 23121920) 23121920, 4096 00020000:00200000:1.0:1789760737.511819:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760737.511838:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5644, 5644] 00000008:00010000:1.0:1789760737.511855:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760737.511861:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760737.511874:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23117824, 23121920) append 0 append lockpos 0 00000080:00200000:1.0:1789760737.511888:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23117824=0x160c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760737.511955:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760737.511958:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23121920 00000020:00010000:1.0:1789760737.511966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760737.511969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760737.511972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760737.511975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760737.511978:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760737.511992:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760737.511998:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760737.512013:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760737.512139:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 23117824<>23121920 00000100:00100000:1.0:1789760737.512216:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760737.512234:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495802368:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.543594:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972961 gen 2 00000008:00010000:1.0:1789760737.543685:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760737.543695:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.543704:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.543733:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760737.543763:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495802368:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760737.543932:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5644, 5644] 00000080:00200000:2.0:1789760737.543948:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.543954:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.543980:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23121920 i_blocks 47111 00000020:00200000:2.0:1789760737.543984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.543994:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.544030:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23121920, count: 4096 00000080:00200000:2.0:1789760737.544038:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23121920, bytes: 4096 00000080:00200000:2.0:1789760737.544047:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.544055:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.544058:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23121920, 23126016) 0 0 00000080:00200000:2.0:1789760737.544062:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.544078:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5645, 5645] 00020000:00200000:2.0:1789760737.544087:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23121920 chunk: [23121920, 23126016) 23126016, 4096 00020000:00200000:2.0:1789760737.544091:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.544111:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5645, 5645] 00000008:00010000:2.0:1789760737.544129:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.544135:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.544146:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23121920, 23126016) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.544161:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23121920=0x160d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.544252:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.544255:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23126016 00000020:00010000:2.0:1789760737.544264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.544267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.544270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.544273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.544276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.544284:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.544290:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.544308:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760737.544733:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 23121920<>23126016 00000100:00100000:2.0:1789760737.544794:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760737.544815:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495802496:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760737.577095:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972962 gen 2 00000008:00010000:2.0:1789760737.577152:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760737.577161:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760737.577214:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760737.577293:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760737.577322:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495802496:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760737.577496:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5645, 5645] 00000080:00200000:3.0:1789760737.577512:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760737.577519:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760737.577535:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23126016 i_blocks 47111 00000020:00200000:3.0:1789760737.577539:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760737.577550:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760737.577588:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23126016, count: 4096 00000080:00200000:3.0:1789760737.577596:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23126016, bytes: 4096 00000080:00200000:3.0:1789760737.577608:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760737.577628:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760737.577631:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23126016, 23130112) 0 0 00000080:00200000:3.0:1789760737.577644:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760737.577662:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5646, 5646] 00020000:00200000:3.0:1789760737.577671:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23126016 chunk: [23126016, 23130112) 23130112, 4096 00020000:00200000:3.0:1789760737.577676:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760737.577696:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5646, 5646] 00000008:00010000:3.0:1789760737.577714:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760737.577720:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760737.577742:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23126016, 23130112) append 0 append lockpos 0 00000080:00200000:3.0:1789760737.577756:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23126016=0x160e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760737.577809:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760737.577812:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23130112 00000020:00010000:3.0:1789760737.577821:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760737.577825:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760737.577828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760737.577831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760737.577834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760737.577841:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760737.578069:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760737.578091:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760737.579025:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 23126016<>23130112 00000100:00100000:2.0:1789760737.579121:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495802624:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760737.624953:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972963 gen 2 00000008:00010000:2.0:1789760737.625005:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:2.0:1789760737.625013:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760737.625024:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760737.625093:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:2.0:1789760737.625122:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495802624:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760737.625382:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5646, 5646] 00000080:00200000:3.0:1789760737.625455:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760737.625462:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.625566:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23130112 i_blocks 47111 00000020:00200000:2.0:1789760737.625570:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.625579:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.625613:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23130112, count: 4096 00000080:00200000:2.0:1789760737.625622:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23130112, bytes: 4096 00000080:00200000:2.0:1789760737.625632:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.625649:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.625652:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23130112, 23134208) 0 0 00000080:00200000:2.0:1789760737.625657:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.625674:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5647, 5647] 00020000:00200000:2.0:1789760737.625682:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23130112 chunk: [23130112, 23134208) 23134208, 4096 00020000:00200000:2.0:1789760737.625687:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.625707:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5647, 5647] 00000008:00010000:2.0:1789760737.625799:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.625806:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760737.625819:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23130112, 23134208) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.625834:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23130112=0x160f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.625947:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.625950:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23134208 00000020:00010000:2.0:1789760737.625959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.625963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.625966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.625970:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.625973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.625981:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.625988:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.626004:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760737.626124:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 23130112<>23134208 00000100:00100000:3.0:1789760737.626231:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495802752:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760737.680921:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972964 gen 2 00000008:00010000:3.0:1789760737.680980:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760737.680989:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.681001:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.681023:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760737.681053:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495802752:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760737.718891:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760737.718907:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.718919:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.718941:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760737.719574:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495802880:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.720703:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803008:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760737.739488:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760737.745419:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972966 gen 2 00000008:00010000:1.0:1789760737.745483:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760737.745492:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.745501:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.745567:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760737.745594:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803008:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.746943:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495803136:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.780106:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972967 gen 2 00000008:00010000:1.0:1789760737.780159:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760737.781191:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.781208:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.781236:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760737.781273:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495803136:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.782553:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803264:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.817546:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972968 gen 2 00000008:00010000:1.0:1789760737.817594:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760737.817602:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.817613:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.817650:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760737.817681:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803264:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.818830:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760737.818864:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803392:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760737.863351:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760737.863367:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.863380:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.863400:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760737.863432:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803392:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.864563:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803520:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.902946:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972970 gen 2 00000008:00010000:1.0:1789760737.903425:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760737.903433:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.903443:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.903467:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760737.903499:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803520:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.904755:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760737.904776:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803648:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.959249:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972971 gen 2 00000008:00010000:1.0:1789760737.959301:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760737.959311:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.959335:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.959359:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760737.959390:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803648:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760737.960854:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803776:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.004573:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972972 gen 2 00000008:00010000:1.0:1789760738.004631:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760738.004651:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.004662:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.004825:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760738.004849:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495803776:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.005556:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495803904:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.037993:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972973 gen 2 00000008:00010000:1.0:1789760738.038070:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760738.038080:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.038091:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.038114:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760738.038153:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495803904:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.040029:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495804032:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.090620:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972974 gen 2 00000008:00010000:1.0:1789760738.090680:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760738.090690:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.090701:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.090725:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760738.091503:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495804032:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760738.091546:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:1.0:1789760738.091688:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 23175168<>23179264 00000100:00100000:1.0:1789760738.091721:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760738.091735:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495804160:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760738.148043:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760738.148068:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.148081:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.148099:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760738.148473:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495804160:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760738.148503:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:1.0:1789760738.148618:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 23179264<>23183360 00000100:00100000:1.0:1789760738.148679:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495804288:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.187663:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495804544:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760738.187933:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1789760738.187942:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495804416:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760738.188056:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495804672:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760738.204683:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972976 gen 2 00000008:00010000:1.0:1789760738.204840:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760738.204848:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.204859:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.204881:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760738.204910:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495804288:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.205815:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760738.205835:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495804800:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760738.257995:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760738.258012:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.258024:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.258045:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760738.258075:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495804800:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760738.258824:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:1.0:1789760738.258950:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 23187456<>23191552 00000100:00100000:1.0:1789760738.259009:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495804928:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.328887:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972978 gen 2 00000008:00010000:1.0:1789760738.328964:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760738.328972:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.328981:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.329002:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760738.329032:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495804928:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.381285:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972979 gen 2 00000008:00010000:1.0:1789760738.381341:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760738.381350:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.381362:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.381392:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760738.381423:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495805056:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760738.381658:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5662, 5662] 00000080:00200000:2.0:1789760738.381675:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.381682:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.381698:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23195648 i_blocks 47111 00000020:00200000:2.0:1789760738.381703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.381713:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.381753:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23195648, count: 4096 00000080:00200000:2.0:1789760738.381762:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23195648, bytes: 4096 00000080:00200000:2.0:1789760738.381773:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.381783:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.381786:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23195648, 23199744) 0 0 00000080:00200000:2.0:1789760738.381791:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.381809:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5663, 5663] 00020000:00200000:2.0:1789760738.381820:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23195648 chunk: [23195648, 23199744) 23199744, 4096 00020000:00200000:2.0:1789760738.381825:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.381858:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5663, 5663] 00000008:00010000:2.0:1789760738.381878:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.381884:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760738.381896:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23195648, 23199744) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.381910:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23195648=0x161f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.382088:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.382092:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23199744 00000020:00010000:2.0:1789760738.382102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.382106:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.382109:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.382113:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.382116:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.382124:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.382132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.382155:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760738.382386:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 23195648<>23199744 00000100:00100000:3.0:1789760738.382472:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495805184:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760738.427712:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972980 gen 2 00000008:00010000:3.0:1789760738.427778:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760738.427788:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.427801:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.427828:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760738.427867:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495805184:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760738.428409:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5663, 5663] 00000080:00200000:1.0:1789760738.428426:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760738.428432:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760738.428449:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23199744 i_blocks 47111 00000020:00200000:1.0:1789760738.428453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760738.428463:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760738.428498:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23199744, count: 4096 00000080:00200000:1.0:1789760738.428505:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23199744, bytes: 4096 00000080:00200000:1.0:1789760738.428516:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760738.428525:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760738.428528:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23199744, 23203840) 0 0 00000080:00200000:1.0:1789760738.428532:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760738.428549:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5664, 5664] 00020000:00200000:1.0:1789760738.428558:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23199744 chunk: [23199744, 23203840) 23203840, 4096 00020000:00200000:1.0:1789760738.428562:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760738.428584:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5664, 5664] 00000008:00010000:1.0:1789760738.428602:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760738.428608:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760738.428621:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23199744, 23203840) append 0 append lockpos 0 00000080:00200000:1.0:1789760738.428648:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23199744=0x1620000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760738.428708:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760738.428711:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23203840 00000020:00010000:1.0:1789760738.428719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760738.428723:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.428727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.428730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760738.428732:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760738.428739:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760738.428746:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760738.428763:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760738.429488:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 23199744<>23203840 00000100:00100000:1.0:1789760738.429540:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760738.429562:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495805312:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.492266:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972981 gen 2 00000008:00010000:1.0:1789760738.492314:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760738.492322:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.492334:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.492367:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760738.492396:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495805312:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.495607:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495805440:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.553250:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972982 gen 2 00000008:00010000:1.0:1789760738.553299:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760738.553308:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.553326:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.553350:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760738.553658:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495805440:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.555671:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495805568:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.611720:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972983 gen 2 00000008:00010000:1.0:1789760738.611770:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760738.611778:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.611790:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.611813:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760738.611845:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495805568:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.612972:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760738.612994:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495805696:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760738.683032:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760738.683051:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.683064:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.683131:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760738.683729:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495805696:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.693925:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760738.693952:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495805824:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.739957:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972985 gen 2 00000008:00010000:1.0:1789760738.740010:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760738.740016:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.740024:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.740041:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760738.740062:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495805824:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.740647:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760738.740661:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495805952:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.806418:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972986 gen 2 00000008:00010000:1.0:1789760738.806470:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760738.806479:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.806490:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.806509:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760738.806538:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495805952:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.807429:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760738.807447:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495806080:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.847021:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972987 gen 2 00000008:00010000:1.0:1789760738.847073:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760738.847081:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.847091:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.847103:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760738.847144:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495806080:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.853198:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760738.853242:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495806208:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.887881:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972988 gen 2 00000008:00010000:1.0:1789760738.887932:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760738.887941:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.888127:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.888148:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760738.888227:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495806208:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.889193:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760738.889212:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495806336:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.922441:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972989 gen 2 00000008:00010000:1.0:1789760738.922494:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760738.922502:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.922514:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.922585:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760738.922635:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495806336:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760738.924162:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495806464:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760738.955277:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760738.955293:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.955305:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.955325:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760738.955355:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495806464:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760738.955463:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5673, 5673] 00000080:00200000:2.0:1789760738.955480:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.955486:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.955501:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23240704 i_blocks 47111 00000020:00200000:2.0:1789760738.955504:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.955512:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.955546:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23240704, count: 4096 00000080:00200000:2.0:1789760738.955553:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23240704, bytes: 4096 00000080:00200000:2.0:1789760738.955563:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.955571:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.955574:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23240704, 23244800) 0 0 00000080:00200000:2.0:1789760738.955578:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.955594:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5674, 5674] 00020000:00200000:2.0:1789760738.955602:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23240704 chunk: [23240704, 23244800) 23244800, 4096 00020000:00200000:2.0:1789760738.955607:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.955636:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5674, 5674] 00000008:00010000:2.0:1789760738.955653:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.955659:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760738.955670:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23240704, 23244800) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.955682:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23240704=0x162a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.955736:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.955739:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23244800 00000020:00010000:2.0:1789760738.955747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.955750:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.955753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.955756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.955758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.955765:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.955771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.955785:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760738.955911:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 23240704<>23244800 00000100:00100000:3.0:1789760738.955986:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495806592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760738.995373:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972991 gen 2 00000008:00010000:3.0:1789760738.995443:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760738.995451:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.995462:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.995485:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760738.995515:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495806592:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760738.995581:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5674, 5674] 00000080:00200000:1.0:1789760738.995598:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760738.995604:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760738.995620:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23244800 i_blocks 47111 00000020:00200000:1.0:1789760738.995624:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760738.995644:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760738.995678:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23244800, count: 4096 00000080:00200000:1.0:1789760738.995685:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23244800, bytes: 4096 00000080:00200000:1.0:1789760738.995695:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760738.995702:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760738.995705:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23244800, 23248896) 0 0 00000080:00200000:1.0:1789760738.995709:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760738.995726:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5675, 5675] 00020000:00200000:1.0:1789760738.995735:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23244800 chunk: [23244800, 23248896) 23248896, 4096 00020000:00200000:1.0:1789760738.995740:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760738.995759:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5675, 5675] 00000008:00010000:1.0:1789760738.995776:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760738.995782:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760738.995793:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23244800, 23248896) append 0 append lockpos 0 00000080:00200000:1.0:1789760738.995807:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23244800=0x162b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760738.995865:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760738.995869:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23248896 00000020:00010000:1.0:1789760738.995877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760738.995880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.995883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.995887:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760738.995890:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760738.995897:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760738.995903:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760738.995920:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760738.996049:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 23244800<>23248896 00000100:00100000:1.0:1789760738.996093:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760738.996108:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495806720:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760739.034875:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760739.034890:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.034901:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.034932:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760739.034964:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495806720:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760739.035072:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5675, 5675] 00000080:00200000:2.0:1789760739.035089:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.035095:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.035112:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23248896 i_blocks 47111 00000020:00200000:2.0:1789760739.035116:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.035125:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.035159:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23248896, count: 4096 00000080:00200000:2.0:1789760739.035166:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23248896, bytes: 4096 00000080:00200000:2.0:1789760739.035234:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.035244:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.035246:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23248896, 23252992) 0 0 00000080:00200000:2.0:1789760739.035251:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.035268:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5676, 5676] 00020000:00200000:2.0:1789760739.035278:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23248896 chunk: [23248896, 23252992) 23252992, 4096 00020000:00200000:2.0:1789760739.035282:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.035303:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5676, 5676] 00000008:00010000:2.0:1789760739.035323:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.035330:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.035342:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23248896, 23252992) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.035357:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23248896=0x162c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.035413:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.035415:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23252992 00000020:00010000:2.0:1789760739.035424:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.035427:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.035430:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.035433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.035436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.035444:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.035450:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.035463:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760739.035664:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 23248896<>23252992 00000100:00100000:3.0:1789760739.035789:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495806848:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760739.072097:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972993 gen 2 00000008:00010000:3.0:1789760739.072162:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760739.072219:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.072232:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.072255:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760739.072276:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495806848:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760739.072663:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5676, 5676] 00000080:00200000:2.0:1789760739.072681:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.072687:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.072704:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23252992 i_blocks 47111 00000020:00200000:2.0:1789760739.072708:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.072716:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.072750:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23252992, count: 4096 00000080:00200000:2.0:1789760739.072758:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23252992, bytes: 4096 00000080:00200000:2.0:1789760739.072769:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.072778:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.072780:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23252992, 23257088) 0 0 00000080:00200000:2.0:1789760739.072785:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.072801:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5677, 5677] 00020000:00200000:2.0:1789760739.072811:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23252992 chunk: [23252992, 23257088) 23257088, 4096 00020000:00200000:2.0:1789760739.072816:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.072835:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5677, 5677] 00000008:00010000:2.0:1789760739.072851:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.072857:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.072868:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23252992, 23257088) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.072882:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23252992=0x162d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.072937:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.072940:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23257088 00000020:00010000:2.0:1789760739.072958:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.072961:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.072964:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.072968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.072971:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.072979:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.072985:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.073003:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760739.073129:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 23252992<>23257088 00000100:00100000:3.0:1789760739.073198:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495806976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760739.170094:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972994 gen 2 00000008:00010000:3.0:1789760739.170994:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760739.171006:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.171019:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.171111:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760739.171152:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495806976:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760739.173891:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5677, 5677] 00000080:00200000:2.0:1789760739.174110:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.174116:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.174135:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23257088 i_blocks 47111 00000020:00200000:2.0:1789760739.174141:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.174158:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.174228:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23257088, count: 4096 00000080:00200000:2.0:1789760739.174236:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23257088, bytes: 4096 00000080:00200000:2.0:1789760739.174253:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.174262:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.174272:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23257088, 23261184) 0 0 00000080:00200000:2.0:1789760739.174277:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.174295:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5678, 5678] 00020000:00200000:2.0:1789760739.174305:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23257088 chunk: [23257088, 23261184) 23261184, 4096 00020000:00200000:2.0:1789760739.174309:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.174333:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5678, 5678] 00000008:00010000:2.0:1789760739.174353:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.174358:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.174371:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23257088, 23261184) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.174397:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23257088=0x162e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.174897:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.175085:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23261184 00000020:00010000:2.0:1789760739.175099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.175102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.175105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.175109:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.175112:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.175121:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.175129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.175166:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760739.175786:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23257088<>23261184 00000100:00100000:3.0:1789760739.175982:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760739.273874:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972995 gen 2 00000008:00010000:3.0:1789760739.273925:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760739.273933:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.273944:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.277013:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760739.277066:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807104:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760739.282592:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5678, 5678] 00000080:00200000:2.0:1789760739.282633:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.282641:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.282661:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23261184 i_blocks 47111 00000020:00200000:2.0:1789760739.282667:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.282684:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.282718:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23261184, count: 4096 00000080:00200000:2.0:1789760739.282727:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23261184, bytes: 4096 00000080:00200000:2.0:1789760739.282743:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.282753:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.282755:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23261184, 23265280) 0 0 00000080:00200000:2.0:1789760739.282760:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.282777:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5679, 5679] 00020000:00200000:2.0:1789760739.282787:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23261184 chunk: [23261184, 23265280) 23265280, 4096 00020000:00200000:2.0:1789760739.282791:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.282814:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5679, 5679] 00000008:00010000:2.0:1789760739.282833:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.282839:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.282851:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23261184, 23265280) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.282865:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23261184=0x162f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.282937:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.282940:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23265280 00000020:00010000:2.0:1789760739.282949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.282952:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.282955:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.282958:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.282961:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.282969:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.282975:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.283004:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760739.283348:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 23261184<>23265280 00000100:00100000:3.0:1789760739.283931:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760739.283968:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807232:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760739.323927:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760739.323942:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.323965:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.323990:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760739.324021:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807232:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760739.324120:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5679, 5679] 00000080:00200000:2.0:1789760739.324149:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.324155:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.324212:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23265280 i_blocks 47111 00000020:00200000:2.0:1789760739.324216:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.324225:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.324422:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23265280, count: 4096 00000080:00200000:3.0:1789760739.324432:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23265280, bytes: 4096 00000080:00200000:3.0:1789760739.324445:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.324453:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.324456:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23265280, 23269376) 0 0 00000080:00200000:3.0:1789760739.324460:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.324479:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5680, 5680] 00020000:00200000:3.0:1789760739.324488:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23265280 chunk: [23265280, 23269376) 23269376, 4096 00020000:00200000:3.0:1789760739.324492:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.324512:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5680, 5680] 00000008:00010000:3.0:1789760739.324530:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.324535:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.324546:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23265280, 23269376) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.324562:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23265280=0x1630000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.324818:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.324821:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23269376 00000020:00010000:3.0:1789760739.324830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.324834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.324837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218250ab8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.324840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.324843:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.324906:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.324913:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.324929:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760739.325111:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 23265280<>23269376 00000100:00100000:2.0:1789760739.325214:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807360:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.360634:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972997 gen 2 00000008:00010000:2.0:1789760739.360708:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760739.360717:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.360727:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.360743:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760739.360775:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807360:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.360776:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5680, 5680] 00000080:00200000:3.0:1789760739.360796:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.360802:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.360883:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23269376 i_blocks 47111 00000020:00200000:3.0:1789760739.360888:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.360898:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.360932:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23269376, count: 4096 00000080:00200000:3.0:1789760739.360939:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23269376, bytes: 4096 00000080:00200000:3.0:1789760739.361208:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.361220:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.361222:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23269376, 23273472) 0 0 00000080:00200000:3.0:1789760739.361227:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.361425:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5681, 5681] 00020000:00200000:3.0:1789760739.361436:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23269376 chunk: [23269376, 23273472) 23273472, 4096 00020000:00200000:3.0:1789760739.361440:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.361464:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5681, 5681] 00000008:00010000:3.0:1789760739.361483:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.361488:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.361500:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23269376, 23273472) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.361516:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23269376=0x1631000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.361579:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.361583:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23273472 00000020:00010000:3.0:1789760739.361592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.361595:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.361598:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218250ab8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.361611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.361614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.361634:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.361643:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.361673:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760739.361806:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 23269376<>23273472 00000100:00100000:2.0:1789760739.362804:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760739.362837:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807488:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.421919:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972998 gen 2 00000008:00010000:2.0:1789760739.421969:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:2.0:1789760739.421977:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.421988:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.422012:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:2.0:1789760739.422040:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807488:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.424249:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5681, 5681] 00000080:00200000:3.0:1789760739.424278:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.424285:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.424306:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23273472 i_blocks 47111 00000020:00200000:3.0:1789760739.424312:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.424328:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.424374:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23273472, count: 4096 00000080:00200000:3.0:1789760739.424383:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23273472, bytes: 4096 00000080:00200000:3.0:1789760739.424398:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.424407:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.424409:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23273472, 23277568) 0 0 00000080:00200000:3.0:1789760739.424414:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.424432:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5682, 5682] 00020000:00200000:3.0:1789760739.424440:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23273472 chunk: [23273472, 23277568) 23277568, 4096 00020000:00200000:3.0:1789760739.424444:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.424467:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5682, 5682] 00000008:00010000:3.0:1789760739.424486:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.424492:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.424504:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23273472, 23277568) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.424520:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23273472=0x1632000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.424589:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.424592:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23277568 00000020:00010000:3.0:1789760739.424600:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.424604:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.424607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218250ab8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.424610:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.424614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.424633:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.424640:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.424662:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760739.424817:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 23273472<>23277568 00000100:00100000:2.0:1789760739.425068:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807616:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.464843:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294972999 gen 2 00000008:00010000:2.0:1789760739.464896:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:2.0:1789760739.464906:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.464917:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.465027:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:2.0:1789760739.465070:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495807616:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.466336:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5682, 5682] 00000080:00200000:3.0:1789760739.466364:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.466371:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.466388:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23277568 i_blocks 47111 00000020:00200000:3.0:1789760739.466392:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.466405:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.466685:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23277568, count: 4096 00000080:00200000:3.0:1789760739.466695:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23277568, bytes: 4096 00000080:00200000:3.0:1789760739.466706:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.466715:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.466717:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23277568, 23281664) 0 0 00000080:00200000:3.0:1789760739.466722:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.466870:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5683, 5683] 00020000:00200000:3.0:1789760739.466878:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23277568 chunk: [23277568, 23281664) 23281664, 4096 00020000:00200000:3.0:1789760739.466883:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.466906:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5683, 5683] 00000008:00010000:3.0:1789760739.466923:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.466928:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.466941:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23277568, 23281664) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.466956:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23277568=0x1633000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.467099:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.467255:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23281664 00000020:00010000:3.0:1789760739.467265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.467269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.467273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218250ab8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.467276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ced58 (ffff8dd218376f38/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.467278:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.467286:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.467293:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.467320:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760739.467447:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 23277568<>23281664 00000100:00100000:2.0:1789760739.467543:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495807744:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.501953:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973000 gen 2 00000008:00010000:2.0:1789760739.502006:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:2.0:1789760739.502015:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.502026:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.502044:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:2.0:1789760739.502079:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495807744:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.502081:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5683, 5683] 00000080:00200000:3.0:1789760739.502103:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.502109:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.502127:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23281664 i_blocks 47111 00000020:00200000:3.0:1789760739.502132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.502251:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.502446:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23281664, count: 4096 00000080:00200000:2.0:1789760739.502454:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23281664, bytes: 4096 00000080:00200000:2.0:1789760739.502467:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.502476:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.502478:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23281664, 23285760) 0 0 00000080:00200000:2.0:1789760739.502484:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.502502:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5684, 5684] 00020000:00200000:2.0:1789760739.502511:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23281664 chunk: [23281664, 23285760) 23285760, 4096 00020000:00200000:2.0:1789760739.502516:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.502535:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5684, 5684] 00000008:00010000:2.0:1789760739.502553:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.502559:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.502572:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23281664, 23285760) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.502586:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23281664=0x1634000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.502722:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.502725:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23285760 00000020:00010000:2.0:1789760739.502734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.502738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.502741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.502745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce7b8 (ffff8dd218376e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.502748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd218376e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.502755:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.502763:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.502827:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760739.503060:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 23281664<>23285760 00000100:00100000:2.0:1789760739.503102:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760739.503119:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495807872:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760739.537322:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760739.537338:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.537350:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.537371:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760739.537398:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495807872:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.537500:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5684, 5684] 00000080:00200000:3.0:1789760739.537517:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.537523:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.537537:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23285760 i_blocks 47111 00000020:00200000:3.0:1789760739.537541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.537550:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.537633:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23285760, count: 4096 00000080:00200000:3.0:1789760739.537642:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23285760, bytes: 4096 00000080:00200000:3.0:1789760739.537651:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.537660:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.537662:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23285760, 23289856) 0 0 00000080:00200000:3.0:1789760739.537666:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.537684:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5685, 5685] 00020000:00200000:3.0:1789760739.537693:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23285760 chunk: [23285760, 23289856) 23289856, 4096 00020000:00200000:3.0:1789760739.537698:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.537718:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5685, 5685] 00000008:00010000:3.0:1789760739.537737:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.537743:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.537756:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23285760, 23289856) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.537770:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23285760=0x1635000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.537828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.537831:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23289856 00000020:00010000:3.0:1789760739.537838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.537842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.537845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.537849:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.537852:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.537860:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.537867:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.537883:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760739.538019:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 23285760<>23289856 00000100:00100000:3.0:1789760739.538061:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760739.538077:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808000:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760739.571641:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973001 gen 2 00000100:00100000:3.0:1789760739.571683:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203147100 x1876699495808000/t4294973002(4294973002) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760755 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760739.571732:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:3.0:1789760739.571739:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.571749:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.571775:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:3.0:1789760739.571802:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808000:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760739.571958:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5685, 5685] 00000080:00200000:1.0:1789760739.571975:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.571981:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.571997:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23289856 i_blocks 47111 00000020:00200000:1.0:1789760739.572001:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.572011:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.572048:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23289856, count: 4096 00000080:00200000:1.0:1789760739.572055:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23289856, bytes: 4096 00000080:00200000:1.0:1789760739.572066:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.572075:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.572077:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23289856, 23293952) 0 0 00000080:00200000:1.0:1789760739.572082:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.572098:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5686, 5686] 00020000:00200000:1.0:1789760739.572107:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23289856 chunk: [23289856, 23293952) 23293952, 4096 00020000:00200000:1.0:1789760739.572112:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.572132:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5686, 5686] 00000008:00010000:1.0:1789760739.572150:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.572155:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760739.572200:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23289856, 23293952) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.572216:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23289856=0x1636000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.572273:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.572276:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23293952 00000020:00010000:1.0:1789760739.572285:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.572289:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.572292:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.572295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.572298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.572306:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.572314:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.572329:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760739.572449:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 23289856<>23293952 00000100:00100000:1.0:1789760739.607906:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973003 gen 2 00000008:00010000:1.0:1789760739.608050:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760739.608058:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.608068:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.608090:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760739.608118:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495808128:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.608245:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5686, 5686] 00000080:00200000:3.0:1789760739.608261:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.608267:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.608283:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23293952 i_blocks 47111 00000020:00200000:3.0:1789760739.608286:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.608295:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.608328:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23293952, count: 4096 00000080:00200000:3.0:1789760739.608335:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23293952, bytes: 4096 00000080:00200000:3.0:1789760739.608344:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.608353:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.608356:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23293952, 23298048) 0 0 00000080:00200000:3.0:1789760739.608361:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.608376:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5687, 5687] 00020000:00200000:3.0:1789760739.608383:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23293952 chunk: [23293952, 23298048) 23298048, 4096 00020000:00200000:3.0:1789760739.608388:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.608406:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5687, 5687] 00000008:00010000:3.0:1789760739.608422:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.608427:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.608437:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23293952, 23298048) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.608451:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23293952=0x1637000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.608500:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.608503:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23298048 00000020:00010000:3.0:1789760739.608510:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.608514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.608517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.608520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.608523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.608530:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.608536:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.608549:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760739.608743:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 23293952<>23298048 00000100:00100000:2.0:1789760739.608860:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808256:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.640510:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973004 gen 2 00000008:00010000:2.0:1789760739.640566:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:2.0:1789760739.640575:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.640585:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.640617:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:2.0:1789760739.640661:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808256:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.640816:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5687, 5687] 00000080:00200000:3.0:1789760739.640834:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.640840:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.640857:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23298048 i_blocks 47111 00000020:00200000:3.0:1789760739.640861:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.640870:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.640904:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23298048, count: 4096 00000080:00200000:3.0:1789760739.640912:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23298048, bytes: 4096 00000080:00200000:3.0:1789760739.640922:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.640931:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.640933:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23298048, 23302144) 0 0 00000080:00200000:3.0:1789760739.640938:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.640964:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5688, 5688] 00020000:00200000:3.0:1789760739.640973:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23298048 chunk: [23298048, 23302144) 23302144, 4096 00020000:00200000:3.0:1789760739.640977:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.640999:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5688, 5688] 00000008:00010000:3.0:1789760739.641018:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.641024:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.641035:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23298048, 23302144) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.641050:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23298048=0x1638000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.641108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.641111:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23302144 00000020:00010000:3.0:1789760739.641120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.641124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.641127:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.641131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.641134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.641141:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.641148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.641165:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760739.641332:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 23298048<>23302144 00000100:00100000:2.0:1789760739.641514:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760739.641533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808384:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.684078:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973005 gen 2 00000008:00010000:2.0:1789760739.684131:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:2.0:1789760739.684140:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.684152:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.684352:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:2.0:1789760739.684384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808384:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.684530:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5688, 5688] 00000080:00200000:3.0:1789760739.684547:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.684554:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.684697:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23302144 i_blocks 47111 00000020:00200000:2.0:1789760739.684702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.684712:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.684749:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23302144, count: 4096 00000080:00200000:2.0:1789760739.684755:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23302144, bytes: 4096 00000080:00200000:2.0:1789760739.684764:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.684771:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.684773:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23302144, 23306240) 0 0 00000080:00200000:2.0:1789760739.684777:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.684792:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5689, 5689] 00020000:00200000:2.0:1789760739.684800:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23302144 chunk: [23302144, 23306240) 23306240, 4096 00020000:00200000:2.0:1789760739.684804:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.684823:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5689, 5689] 00000008:00010000:2.0:1789760739.684839:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.684843:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.684861:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23302144, 23306240) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.684874:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23302144=0x1639000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.684917:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.684919:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23306240 00000020:00010000:2.0:1789760739.684925:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.684928:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.684930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.684933:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.684935:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.684940:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.684954:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.684992:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760739.685115:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 23302144<>23306240 00000100:00100000:2.0:1789760739.685164:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760739.685211:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495808512:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.709371:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973006 gen 2 00000008:00010000:2.0:1789760739.709418:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760739.709424:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.709433:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.709454:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760739.709481:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495808512:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.709494:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5689, 5689] 00000080:00200000:3.0:1789760739.709514:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.709520:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.709537:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23306240 i_blocks 47111 00000020:00200000:3.0:1789760739.709540:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.709552:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.709707:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23306240, count: 4096 00000080:00200000:2.0:1789760739.709716:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23306240, bytes: 4096 00000080:00200000:2.0:1789760739.709726:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.709735:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.709738:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23306240, 23310336) 0 0 00000080:00200000:2.0:1789760739.709743:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.709761:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5690, 5690] 00020000:00200000:2.0:1789760739.709770:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23306240 chunk: [23306240, 23310336) 23310336, 4096 00020000:00200000:2.0:1789760739.709775:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.709824:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5690, 5690] 00000008:00010000:2.0:1789760739.709844:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.709921:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.709935:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23306240, 23310336) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.710016:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23306240=0x163a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.710075:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.710079:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23310336 00000020:00010000:2.0:1789760739.710089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.710092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.710096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.710099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.710102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.710110:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.710116:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.710151:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760739.710313:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 23306240<>23310336 00000100:00100000:3.0:1789760739.710428:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760739.710445:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808640:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760739.763817:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973007 gen 2 00000008:00010000:3.0:1789760739.763869:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760739.763877:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.763889:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.763975:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760739.764009:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808640:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760739.765705:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5690, 5690] 00000080:00200000:2.0:1789760739.765729:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.765736:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.765754:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23310336 i_blocks 47111 00000020:00200000:2.0:1789760739.765758:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.765771:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.765805:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23310336, count: 4096 00000080:00200000:2.0:1789760739.765812:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23310336, bytes: 4096 00000080:00200000:2.0:1789760739.765825:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.765834:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.765836:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23310336, 23314432) 0 0 00000080:00200000:2.0:1789760739.765842:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.765859:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5691, 5691] 00020000:00200000:2.0:1789760739.765867:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23310336 chunk: [23310336, 23314432) 23314432, 4096 00020000:00200000:2.0:1789760739.765872:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.765892:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5691, 5691] 00000008:00010000:2.0:1789760739.765911:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.765916:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.765928:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23310336, 23314432) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.765942:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23310336=0x163b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.766009:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.766013:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23314432 00000020:00010000:2.0:1789760739.766022:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.766025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.766029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.766032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.766035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.766045:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.766051:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.766079:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760739.766479:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23310336<>23314432 00000100:00100000:3.0:1789760739.766584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808768:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760739.790922:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760739.808397:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973008 gen 2 00000008:00010000:2.0:1789760739.808629:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:2.0:1789760739.808638:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.808649:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.808697:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000080:00200000:1.0:1789760739.808715:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5691, 5691] 00000100:00100000:2.0:1789760739.808727:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495808768:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760739.808733:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.808738:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.808753:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23314432 i_blocks 47111 00000020:00200000:1.0:1789760739.808756:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.808765:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.808800:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23314432, count: 4096 00000080:00200000:1.0:1789760739.808807:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23314432, bytes: 4096 00000080:00200000:1.0:1789760739.808816:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.808825:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.808828:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23314432, 23318528) 0 0 00000080:00200000:1.0:1789760739.808833:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.808908:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5692, 5692] 00020000:00200000:1.0:1789760739.808918:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23314432 chunk: [23314432, 23318528) 23318528, 4096 00020000:00200000:1.0:1789760739.808922:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.808944:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5692, 5692] 00000008:00010000:1.0:1789760739.808961:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.808967:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760739.808979:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23314432, 23318528) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.808994:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23314432=0x163c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.809048:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.809051:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23318528 00000020:00010000:1.0:1789760739.809059:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.809063:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.809067:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.809070:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.809074:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.809082:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.809088:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.809104:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760739.809299:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 23314432<>23318528 00000008:00010000:1.0:1789760739.847710:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760739.847725:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.847735:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.847772:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760739.847804:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495808896:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760739.848146:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5692, 5692] 00000080:00200000:2.0:1789760739.848164:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.848219:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.848236:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23318528 i_blocks 47111 00000020:00200000:2.0:1789760739.848239:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.848250:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.848290:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23318528, count: 4096 00000080:00200000:2.0:1789760739.848297:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23318528, bytes: 4096 00000080:00200000:2.0:1789760739.848309:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.848317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.848320:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23318528, 23322624) 0 0 00000080:00200000:2.0:1789760739.848325:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.848341:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5693, 5693] 00020000:00200000:2.0:1789760739.848350:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23318528 chunk: [23318528, 23322624) 23322624, 4096 00020000:00200000:2.0:1789760739.848354:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.848373:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5693, 5693] 00000008:00010000:2.0:1789760739.848391:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.848397:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.848408:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23318528, 23322624) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.848422:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23318528=0x163d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.848479:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.848482:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23322624 00000020:00010000:2.0:1789760739.848491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.848494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.848497:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.848500:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.848503:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.848511:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.848519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.848536:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760739.848679:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23318528<>23322624 00000100:00100000:2.0:1789760739.848719:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760739.848734:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495809024:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.896029:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973010 gen 2 00000008:00010000:2.0:1789760739.896093:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:2.0:1789760739.896103:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.896113:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.896136:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:2.0:1789760739.896166:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495809024:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.896354:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5693, 5693] 00000080:00200000:3.0:1789760739.896398:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.896405:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.896419:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23322624 i_blocks 47111 00000020:00200000:3.0:1789760739.896423:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.896431:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.896475:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23322624, count: 4096 00000080:00200000:3.0:1789760739.896483:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23322624, bytes: 4096 00000080:00200000:3.0:1789760739.896494:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.896502:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.896505:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23322624, 23326720) 0 0 00000080:00200000:3.0:1789760739.896509:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.896525:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5694, 5694] 00020000:00200000:3.0:1789760739.896533:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23322624 chunk: [23322624, 23326720) 23326720, 4096 00020000:00200000:3.0:1789760739.896538:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.896558:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5694, 5694] 00000008:00010000:3.0:1789760739.896575:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.896581:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.896593:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23322624, 23326720) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.896689:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23322624=0x163e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.896751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.896754:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23326720 00000020:00010000:3.0:1789760739.896762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.896765:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.896768:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.896772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.896774:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.896782:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.896790:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.896808:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760739.897079:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 23322624<>23326720 00000100:00100000:3.0:1789760739.897130:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760739.897149:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495809152:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760739.929665:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973011 gen 2 00000008:00010000:3.0:1789760739.929713:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:3.0:1789760739.929721:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.929732:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.929761:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:3.0:1789760739.929800:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495809152:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760739.930405:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5694, 5694] 00000080:00200000:2.0:1789760739.930434:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.930441:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.930457:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23326720 i_blocks 47111 00000020:00200000:2.0:1789760739.930461:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.930471:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.930505:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23326720, count: 4096 00000080:00200000:2.0:1789760739.930513:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23326720, bytes: 4096 00000080:00200000:2.0:1789760739.930525:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.930534:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.930536:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23326720, 23330816) 0 0 00000080:00200000:2.0:1789760739.930540:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.930555:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5695, 5695] 00020000:00200000:2.0:1789760739.930563:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23326720 chunk: [23326720, 23330816) 23330816, 4096 00020000:00200000:2.0:1789760739.930568:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.930599:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5695, 5695] 00000008:00010000:2.0:1789760739.930632:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.930638:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760739.931244:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23326720, 23330816) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.931262:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23326720=0x163f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.931331:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.931334:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23330816 00000020:00010000:2.0:1789760739.931343:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.931347:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.931350:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.931353:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.931357:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.931366:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.931373:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.931395:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760739.931522:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23326720<>23330816 00000100:00100000:2.0:1789760739.931574:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760739.931614:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495809280:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.956253:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973012 gen 2 00000008:00010000:2.0:1789760739.956300:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:2.0:1789760739.956308:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.956320:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.956340:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:2.0:1789760739.956368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495809280:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760739.956479:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5695, 5695] 00000080:00200000:3.0:1789760739.956496:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.956501:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.956517:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23330816 i_blocks 47111 00000020:00200000:3.0:1789760739.956520:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.956530:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.956563:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23330816, count: 4096 00000080:00200000:3.0:1789760739.956570:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23330816, bytes: 4096 00000080:00200000:3.0:1789760739.956580:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.956590:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.956593:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23330816, 23334912) 0 0 00000080:00200000:3.0:1789760739.956597:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.956633:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5696, 5696] 00020000:00200000:3.0:1789760739.956641:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23330816 chunk: [23330816, 23334912) 23334912, 4096 00020000:00200000:3.0:1789760739.956645:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.956664:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5696, 5696] 00000008:00010000:3.0:1789760739.956685:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.956689:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.956700:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23330816, 23334912) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.956714:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23330816=0x1640000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.956769:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.956772:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23334912 00000020:00010000:3.0:1789760739.956780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.956783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.956785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.956789:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.956793:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.956800:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.956806:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.956822:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760739.956946:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 23330816<>23334912 00000100:00100000:2.0:1789760739.957006:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495809408:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760739.984702:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973013 gen 2 00000008:00010000:2.0:1789760739.984750:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:2.0:1789760739.984759:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.984769:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.984791:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000080:00200000:3.0:1789760739.985105:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5696, 5696] 00000080:00200000:3.0:1789760739.985123:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.985129:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.985144:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23334912 i_blocks 47111 00000020:00200000:3.0:1789760739.985148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.985157:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:2.0:1789760739.986521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495809408:192.168.203.109@tcp:4:dd.0 00000080:00200008:3.0:1789760739.986522:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23334912, count: 4096 00000080:00200000:3.0:1789760739.986534:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23334912, bytes: 4096 00000080:00200000:3.0:1789760739.986547:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.986556:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.986558:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23334912, 23339008) 0 0 00000080:00200000:3.0:1789760739.986563:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.986582:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5697, 5697] 00020000:00200000:3.0:1789760739.986590:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23334912 chunk: [23334912, 23339008) 23339008, 4096 00020000:00200000:3.0:1789760739.986595:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.986616:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5697, 5697] 00000008:00010000:3.0:1789760739.986645:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.986650:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760739.986662:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23334912, 23339008) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.986676:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23334912=0x1641000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.986737:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.986740:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23339008 00000020:00010000:3.0:1789760739.986749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.986753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.986756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.986759:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.986762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.986769:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.986776:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.986794:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760739.987119:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 23334912<>23339008 00000100:00100000:3.0:1789760739.987218:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495809536:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760740.037394:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:2.0:1789760740.037411:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.037424:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.037448:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:2.0:1789760740.037478:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495809536:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760740.037571:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5697, 5697] 00000080:00200000:1.0:1789760740.037599:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760740.037605:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760740.037631:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23339008 i_blocks 47111 00000020:00200000:1.0:1789760740.037635:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760740.037645:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760740.037891:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23339008, count: 4096 00000080:00200000:1.0:1789760740.037899:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23339008, bytes: 4096 00000080:00200000:1.0:1789760740.037909:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760740.037918:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760740.037920:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23339008, 23343104) 0 0 00000080:00200000:1.0:1789760740.037924:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760740.037942:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5698, 5698] 00020000:00200000:1.0:1789760740.037950:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23339008 chunk: [23339008, 23343104) 23343104, 4096 00020000:00200000:1.0:1789760740.037954:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760740.037974:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5698, 5698] 00000008:00010000:1.0:1789760740.037994:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760740.038000:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760740.038011:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23339008, 23343104) append 0 append lockpos 0 00000080:00200000:1.0:1789760740.038026:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23339008=0x1642000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760740.038233:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760740.038236:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23343104 00000020:00010000:1.0:1789760740.038245:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760740.038248:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760740.038250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760740.038253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760740.038255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760740.038261:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760740.038267:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760740.038276:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760740.038393:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23339008<>23343104 00000100:00100000:1.0:1789760740.038428:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760740.038443:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495809664:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760740.071403:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973015 gen 2 00000008:00010000:1.0:1789760740.071468:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760740.071477:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.071487:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.071513:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760740.071540:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495809664:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760740.120606:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973016 gen 2 00000008:00010000:1.0:1789760740.120671:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760740.120680:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.120690:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.120711:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760740.120742:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495809792:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760740.155831:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760740.155910:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.155922:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.155991:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760740.156021:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495809920:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.156098:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5700, 5700] 00000080:00200000:2.0:1789760740.156116:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.156122:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.156136:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23351296 i_blocks 47111 00000020:00200000:2.0:1789760740.156139:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.156148:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.156447:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23351296, count: 4096 00000080:00200000:2.0:1789760740.156455:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23351296, bytes: 4096 00000080:00200000:2.0:1789760740.156468:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.156477:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.156480:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23351296, 23355392) 0 0 00000080:00200000:2.0:1789760740.156484:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.156500:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5701, 5701] 00020000:00200000:2.0:1789760740.156508:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23351296 chunk: [23351296, 23355392) 23355392, 4096 00020000:00200000:2.0:1789760740.156513:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.156532:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5701, 5701] 00000008:00010000:2.0:1789760740.156549:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.156555:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.156567:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23351296, 23355392) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.156582:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23351296=0x1645000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.156649:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.156653:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23355392 00000020:00010000:2.0:1789760740.156661:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.156665:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.156668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.156672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.156675:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.156682:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.156689:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.156708:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760740.157011:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23351296<>23355392 00000100:00100000:3.0:1789760740.157107:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495810048:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760740.198546:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973018 gen 2 00000008:00010000:3.0:1789760740.198634:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760740.198642:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.198653:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.198770:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760740.198801:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495810048:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760740.200780:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495810176:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760740.246546:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973019 gen 2 00000008:00010000:1.0:1789760740.246597:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760740.246607:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.246684:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.246706:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760740.246739:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495810176:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760740.313480:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973020 gen 2 00000008:00010000:1.0:1789760740.313539:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760740.313547:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.313558:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.313584:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760740.313633:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495810304:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.313875:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5703, 5703] 00000080:00200000:2.0:1789760740.313893:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.313900:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.313916:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23363584 i_blocks 47111 00000020:00200000:2.0:1789760740.313920:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.313930:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.313966:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23363584, count: 4096 00000080:00200000:2.0:1789760740.313973:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23363584, bytes: 4096 00000080:00200000:2.0:1789760740.313984:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.313993:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.313995:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23363584, 23367680) 0 0 00000080:00200000:2.0:1789760740.314000:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.314017:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5704, 5704] 00020000:00200000:2.0:1789760740.314025:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23363584 chunk: [23363584, 23367680) 23367680, 4096 00020000:00200000:2.0:1789760740.314029:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.314049:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5704, 5704] 00000008:00010000:2.0:1789760740.314066:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.314072:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.314083:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23363584, 23367680) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.314097:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23363584=0x1648000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.314155:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.314159:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23367680 00000020:00010000:2.0:1789760740.314209:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.314213:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.314216:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.314219:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.314222:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.314229:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.314236:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.314252:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760740.314374:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 23363584<>23367680 00000100:00100000:3.0:1789760740.314663:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760740.314684:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495810432:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760740.377088:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760740.377106:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.377120:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.377330:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760740.377366:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495810432:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.377399:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5704, 5704] 00000080:00200000:2.0:1789760740.377416:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.377424:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.377439:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23367680 i_blocks 47111 00000020:00200000:2.0:1789760740.377444:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.377453:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.377546:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23367680, count: 4096 00000080:00200000:2.0:1789760740.377556:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23367680, bytes: 4096 00000080:00200000:2.0:1789760740.377565:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.377574:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.377577:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23367680, 23371776) 0 0 00000080:00200000:2.0:1789760740.377581:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.377599:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5705, 5705] 00020000:00200000:2.0:1789760740.377607:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23367680 chunk: [23367680, 23371776) 23371776, 4096 00020000:00200000:2.0:1789760740.377612:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.377643:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5705, 5705] 00000008:00010000:2.0:1789760740.377662:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.377667:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.377679:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23367680, 23371776) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.377693:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23367680=0x1649000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.377749:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.377752:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23371776 00000020:00010000:2.0:1789760740.377760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.377764:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.377767:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.377770:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.377773:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.377781:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.377788:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.377803:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760740.377930:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 23367680<>23371776 00000100:00100000:3.0:1789760740.378036:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495810560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760740.426220:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973022 gen 2 00000008:00010000:3.0:1789760740.426607:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760740.426626:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.426636:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.426664:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760740.426698:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495810560:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.426974:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5705, 5705] 00000080:00200000:2.0:1789760740.426990:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.426998:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.427015:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23371776 i_blocks 47111 00000020:00200000:2.0:1789760740.427019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.427029:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.427063:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23371776, count: 4096 00000080:00200000:2.0:1789760740.427071:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23371776, bytes: 4096 00000080:00200000:2.0:1789760740.427083:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.427093:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.427095:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23371776, 23375872) 0 0 00000080:00200000:2.0:1789760740.427100:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.427117:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5706, 5706] 00020000:00200000:2.0:1789760740.427125:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23371776 chunk: [23371776, 23375872) 23375872, 4096 00020000:00200000:2.0:1789760740.427130:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.427149:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5706, 5706] 00000008:00010000:2.0:1789760740.427910:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.427919:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.428084:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23371776, 23375872) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.428102:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23371776=0x164a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.428197:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.428201:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23375872 00000020:00010000:2.0:1789760740.428210:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.428214:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.428217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.428220:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.428223:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.428232:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.428239:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.428916:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760740.429118:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 23371776<>23375872 00000100:00100000:2.0:1789760740.429480:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495810688:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760740.462246:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760740.462261:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.462273:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.462303:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760740.462333:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495810688:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760740.462593:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5706, 5706] 00000080:00200000:1.0:1789760740.462611:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760740.462628:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760740.462643:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23375872 i_blocks 47111 00000020:00200000:1.0:1789760740.462647:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760740.462658:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760740.462693:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23375872, count: 4096 00000080:00200000:1.0:1789760740.462700:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23375872, bytes: 4096 00000080:00200000:1.0:1789760740.462710:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760740.462719:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760740.462722:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23375872, 23379968) 0 0 00000080:00200000:1.0:1789760740.462727:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760740.462743:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5707, 5707] 00020000:00200000:1.0:1789760740.462752:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23375872 chunk: [23375872, 23379968) 23379968, 4096 00020000:00200000:1.0:1789760740.462757:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760740.462776:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5707, 5707] 00000008:00010000:1.0:1789760740.462794:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760740.462799:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760740.462810:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23375872, 23379968) append 0 append lockpos 0 00000080:00200000:1.0:1789760740.462824:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23375872=0x164b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760740.462880:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760740.462883:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23379968 00000020:00010000:1.0:1789760740.462892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760740.462896:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760740.462899:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760740.462903:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760740.462906:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760740.462913:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760740.462921:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760740.462936:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760740.463060:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23375872<>23379968 00000100:00100000:1.0:1789760740.499159:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973024 gen 2 00000008:00010000:1.0:1789760740.499266:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760740.499274:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.499284:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.499308:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760740.499337:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495810816:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.499374:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5707, 5707] 00000080:00200000:2.0:1789760740.499392:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.499398:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.499414:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23379968 i_blocks 47111 00000020:00200000:2.0:1789760740.499418:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.499428:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.499461:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23379968, count: 4096 00000080:00200000:2.0:1789760740.499468:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23379968, bytes: 4096 00000080:00200000:2.0:1789760740.499479:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.499487:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.499490:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23379968, 23384064) 0 0 00000080:00200000:2.0:1789760740.499494:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.499512:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5708, 5708] 00020000:00200000:2.0:1789760740.499521:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23379968 chunk: [23379968, 23384064) 23384064, 4096 00020000:00200000:2.0:1789760740.499525:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.499556:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5708, 5708] 00000008:00010000:2.0:1789760740.499576:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.499582:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.499609:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23379968, 23384064) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.499631:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23379968=0x164c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.499688:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.499691:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23384064 00000020:00010000:2.0:1789760740.499701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.499705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.499709:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.499713:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.499717:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.499724:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.499731:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.499758:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760740.499878:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 23379968<>23384064 00000100:00100000:2.0:1789760740.499923:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760740.499949:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495810944:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760740.544957:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973025 gen 2 00000008:00010000:2.0:1789760740.545006:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760740.545079:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.545092:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.545116:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760740.545146:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495810944:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760740.545298:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5708, 5708] 00000080:00200000:3.0:1789760740.545315:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.545322:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.545337:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23384064 i_blocks 47111 00000020:00200000:3.0:1789760740.545340:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.545350:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.545384:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23384064, count: 4096 00000080:00200000:3.0:1789760740.545391:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23384064, bytes: 4096 00000080:00200000:3.0:1789760740.545404:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.545412:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.545414:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23384064, 23388160) 0 0 00000080:00200000:3.0:1789760740.545418:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.545433:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5709, 5709] 00020000:00200000:3.0:1789760740.545440:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23384064 chunk: [23384064, 23388160) 23388160, 4096 00020000:00200000:3.0:1789760740.545444:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.545462:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5709, 5709] 00000008:00010000:3.0:1789760740.545479:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.545485:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760740.545495:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23384064, 23388160) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.545508:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23384064=0x164d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.545554:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.545556:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23388160 00000020:00010000:3.0:1789760740.545564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.545567:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.545570:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.545573:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.545576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.545582:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.545589:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.546472:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:2.0:1789760740.546634:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 23384064<>23388160 00000100:00100000:2.0:1789760740.546701:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495811072:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760740.584969:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973026 gen 2 00000008:00010000:2.0:1789760740.585020:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:2.0:1789760740.585029:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.585041:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.585067:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:2.0:1789760740.585097:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495811072:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760740.585118:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5709, 5709] 00000080:00200000:3.0:1789760740.585137:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.585144:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.585162:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23388160 i_blocks 47111 00000020:00200000:3.0:1789760740.585184:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.585195:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.585229:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23388160, count: 4096 00000080:00200000:3.0:1789760740.585236:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23388160, bytes: 4096 00000080:00200000:3.0:1789760740.585249:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.585258:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.585261:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23388160, 23392256) 0 0 00000080:00200000:3.0:1789760740.585265:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.585281:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5710, 5710] 00020000:00200000:3.0:1789760740.585290:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23388160 chunk: [23388160, 23392256) 23392256, 4096 00020000:00200000:3.0:1789760740.585294:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.585313:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5710, 5710] 00000008:00010000:3.0:1789760740.585330:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.585336:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760740.585348:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23388160, 23392256) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.585363:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23388160=0x164e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.585427:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.585431:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23392256 00000020:00010000:3.0:1789760740.585439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.585442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.585446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.585449:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.585452:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.585460:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.585467:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.585485:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760740.586041:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 23388160<>23392256 00000100:00100000:2.0:1789760740.586259:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495811200:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760740.619726:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:2.0:1789760740.619754:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.619768:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.619790:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:2.0:1789760740.619820:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495811200:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760740.620085:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5710, 5710] 00000080:00200000:3.0:1789760740.620103:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.620110:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.620127:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23392256 i_blocks 47111 00000020:00200000:3.0:1789760740.620131:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.620140:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.620200:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23392256, count: 4096 00000080:00200000:3.0:1789760740.620208:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23392256, bytes: 4096 00000080:00200000:3.0:1789760740.620220:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.620229:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.620231:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23392256, 23396352) 0 0 00000080:00200000:3.0:1789760740.620236:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.620253:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5711, 5711] 00020000:00200000:3.0:1789760740.620261:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23392256 chunk: [23392256, 23396352) 23396352, 4096 00020000:00200000:3.0:1789760740.620266:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.620287:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5711, 5711] 00000008:00010000:3.0:1789760740.620304:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.620310:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760740.620322:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23392256, 23396352) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.620337:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23392256=0x164f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.620391:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.620394:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23396352 00000020:00010000:3.0:1789760740.620402:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.620405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.620408:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.620411:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.620414:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.620421:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.620428:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.620445:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760740.620577:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 23392256<>23396352 00000100:00100000:2.0:1789760740.620820:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760740.620840:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495811328:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760740.654765:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973028 gen 2 00000008:00010000:2.0:1789760740.654834:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:2.0:1789760740.654844:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.654855:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.654880:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:2.0:1789760740.654911:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495811328:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760740.655134:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5711, 5711] 00000080:00200000:3.0:1789760740.655152:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.655159:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.655217:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23396352 i_blocks 47111 00000020:00200000:3.0:1789760740.655221:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.655231:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.655265:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23396352, count: 4096 00000080:00200000:3.0:1789760740.655272:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23396352, bytes: 4096 00000080:00200000:3.0:1789760740.655282:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.655290:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.655293:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23396352, 23400448) 0 0 00000080:00200000:3.0:1789760740.655297:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.655315:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5712, 5712] 00020000:00200000:3.0:1789760740.655324:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23396352 chunk: [23396352, 23400448) 23400448, 4096 00020000:00200000:3.0:1789760740.655329:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.655350:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5712, 5712] 00000008:00010000:3.0:1789760740.655369:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.655375:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760740.655386:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23396352, 23400448) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.655400:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23396352=0x1650000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.655453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.655456:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23400448 00000020:00010000:3.0:1789760740.655465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.655469:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.655472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.655475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.655479:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.655487:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.655493:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.655511:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760740.655658:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 23396352<>23400448 00000100:00100000:2.0:1789760740.655868:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495811456:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760740.696454:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760740.696470:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.696482:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.696505:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760740.696533:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495811456:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760740.696681:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5712, 5712] 00000080:00200000:3.0:1789760740.696698:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.696705:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.696720:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23400448 i_blocks 47111 00000020:00200000:3.0:1789760740.696724:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.696732:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.696773:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23400448, count: 4096 00000080:00200000:3.0:1789760740.696780:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23400448, bytes: 4096 00000080:00200000:3.0:1789760740.696790:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.696799:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.696801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23400448, 23404544) 0 0 00000080:00200000:3.0:1789760740.696805:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.696822:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5713, 5713] 00020000:00200000:3.0:1789760740.696830:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23400448 chunk: [23400448, 23404544) 23404544, 4096 00020000:00200000:3.0:1789760740.696835:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.696865:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5713, 5713] 00000008:00010000:3.0:1789760740.696883:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.696889:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760740.696900:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23400448, 23404544) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.696913:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23400448=0x1651000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.696972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.696974:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23404544 00000020:00010000:3.0:1789760740.696983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.696986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.696990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.696993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.696996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.697002:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.697008:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.697023:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760740.697139:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 23400448<>23404544 00000100:00100000:2.0:1789760740.697400:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760740.697416:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495811584:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760740.728928:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973030 gen 2 00000008:00010000:2.0:1789760740.729004:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:2.0:1789760740.729013:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.729023:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.729048:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:2.0:1789760740.729079:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495811584:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760740.729488:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5713, 5713] 00000080:00200000:1.0:1789760740.729506:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760740.729513:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760740.729530:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23404544 i_blocks 47111 00000020:00200000:1.0:1789760740.729534:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760740.729544:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760740.729576:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23404544, count: 4096 00000080:00200000:1.0:1789760740.729584:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23404544, bytes: 4096 00000080:00200000:1.0:1789760740.729605:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760740.729625:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760740.729627:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23404544, 23408640) 0 0 00000080:00200000:1.0:1789760740.729631:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760740.729649:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5714, 5714] 00020000:00200000:1.0:1789760740.729656:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23404544 chunk: [23404544, 23408640) 23408640, 4096 00020000:00200000:1.0:1789760740.729661:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760740.729681:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5714, 5714] 00000008:00010000:1.0:1789760740.729699:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760740.729705:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760740.729718:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23404544, 23408640) append 0 append lockpos 0 00000080:00200000:1.0:1789760740.729732:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23404544=0x1652000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760740.729903:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760740.729906:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23408640 00000020:00010000:1.0:1789760740.729916:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760740.729920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760740.729923:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760740.729926:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760740.729930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760740.729949:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760740.729955:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760740.729974:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760740.730101:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23404544<>23408640 00000100:00100000:1.0:1789760740.730156:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760740.730216:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495811712:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760740.756818:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973031 gen 2 00000008:00010000:1.0:1789760740.757951:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760740.757962:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.757975:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.758000:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760740.758041:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495811712:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.758056:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5714, 5714] 00000080:00200000:2.0:1789760740.758074:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.758080:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.758095:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23408640 i_blocks 47111 00000020:00200000:2.0:1789760740.758099:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.758108:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.758144:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23408640, count: 4096 00000080:00200000:2.0:1789760740.758151:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23408640, bytes: 4096 00000080:00200000:2.0:1789760740.758162:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.758190:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.758193:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23408640, 23412736) 0 0 00000080:00200000:2.0:1789760740.758198:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.758218:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5715, 5715] 00020000:00200000:2.0:1789760740.758227:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23408640 chunk: [23408640, 23412736) 23412736, 4096 00020000:00200000:2.0:1789760740.758231:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.758252:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5715, 5715] 00000008:00010000:2.0:1789760740.758272:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.758278:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.758291:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23408640, 23412736) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.758307:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23408640=0x1653000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.758365:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.758368:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23412736 00000020:00010000:2.0:1789760740.758377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.758380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.758383:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.758387:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.758390:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.758398:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.758405:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.758421:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760740.758548:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 23408640<>23412736 00000100:00100000:2.0:1789760740.758598:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760740.758625:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495811840:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760740.799669:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973032 gen 2 00000008:00010000:2.0:1789760740.799722:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760740.799731:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.799742:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.799770:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760740.799972:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495811840:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760740.800165:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5715, 5715] 00000080:00200000:1.0:1789760740.800215:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760740.800221:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760740.800237:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23412736 i_blocks 47111 00000020:00200000:1.0:1789760740.800240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760740.800249:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760740.800284:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23412736, count: 4096 00000080:00200000:1.0:1789760740.800291:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23412736, bytes: 4096 00000080:00200000:1.0:1789760740.800303:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760740.800312:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760740.800314:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23412736, 23416832) 0 0 00000080:00200000:1.0:1789760740.800319:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760740.800336:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5716, 5716] 00020000:00200000:1.0:1789760740.800344:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23412736 chunk: [23412736, 23416832) 23416832, 4096 00020000:00200000:1.0:1789760740.800349:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760740.800370:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5716, 5716] 00000008:00010000:1.0:1789760740.800390:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760740.800396:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760740.800408:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23412736, 23416832) append 0 append lockpos 0 00000080:00200000:1.0:1789760740.800422:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23412736=0x1654000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760740.800480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760740.800484:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23416832 00000020:00010000:1.0:1789760740.800493:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760740.800496:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760740.800499:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760740.800503:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760740.800506:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760740.800513:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760740.800520:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760740.800536:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760740.800733:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23412736<>23416832 00010000:00010000:1.0:1789760740.811459:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760740.848627:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973033 gen 2 00000008:00010000:1.0:1789760740.849008:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760740.849017:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.849029:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.849120:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760740.849151:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495811968:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.849443:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5716, 5716] 00000080:00200000:2.0:1789760740.849460:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.849466:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.849481:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23416832 i_blocks 47111 00000020:00200000:2.0:1789760740.849485:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.849495:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.849527:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23416832, count: 4096 00000080:00200000:2.0:1789760740.849533:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23416832, bytes: 4096 00000080:00200000:2.0:1789760740.849545:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.849554:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.849556:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23416832, 23420928) 0 0 00000080:00200000:2.0:1789760740.849560:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.849576:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5717, 5717] 00020000:00200000:2.0:1789760740.849585:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23416832 chunk: [23416832, 23420928) 23420928, 4096 00020000:00200000:2.0:1789760740.849590:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.849608:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5717, 5717] 00000008:00010000:2.0:1789760740.849639:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.849646:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.849656:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23416832, 23420928) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.849670:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23416832=0x1655000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.850131:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.850134:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23420928 00000020:00010000:2.0:1789760740.850144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.850149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.850152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.850155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.850158:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.850188:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.850197:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.850217:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760740.851369:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 23416832<>23420928 00000100:00100000:3.0:1789760740.852490:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760740.852519:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812096:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760740.908669:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760740.908693:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.908706:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.908731:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760740.908762:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812096:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.909268:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5717, 5717] 00000080:00200000:2.0:1789760740.909288:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.909294:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.909309:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23420928 i_blocks 47111 00000020:00200000:2.0:1789760740.909313:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.909322:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.909366:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23420928, count: 4096 00000080:00200000:2.0:1789760740.909374:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23420928, bytes: 4096 00000080:00200000:2.0:1789760740.909387:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.909396:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.909399:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23420928, 23425024) 0 0 00000080:00200000:2.0:1789760740.909404:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.909421:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5718, 5718] 00020000:00200000:2.0:1789760740.909430:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23420928 chunk: [23420928, 23425024) 23425024, 4096 00020000:00200000:2.0:1789760740.909434:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.909455:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5718, 5718] 00000008:00010000:2.0:1789760740.909474:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.909481:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.909493:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23420928, 23425024) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.909508:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23420928=0x1656000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.909566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.909569:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23425024 00000020:00010000:2.0:1789760740.909577:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.909582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.909585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.909590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.909592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.909600:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.909607:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.909634:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760740.909885:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 23420928<>23425024 00000100:00100000:3.0:1789760740.910217:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812224:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760740.957461:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973034 gen 2 00000100:00100000:3.0:1789760740.957503:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495812224/t4294973035(4294973035) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760756 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760740.957549:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760740.957557:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.957568:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.957591:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760740.957623:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812224:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760740.957829:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5718, 5718] 00000080:00200000:2.0:1789760740.957847:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.957854:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.957868:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23425024 i_blocks 47111 00000020:00200000:2.0:1789760740.957872:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.957881:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.957912:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23425024, count: 4096 00000080:00200000:2.0:1789760740.957919:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23425024, bytes: 4096 00000080:00200000:2.0:1789760740.957928:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.957937:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.957940:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23425024, 23429120) 0 0 00000080:00200000:2.0:1789760740.957944:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.957960:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5719, 5719] 00020000:00200000:2.0:1789760740.957968:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23425024 chunk: [23425024, 23429120) 23429120, 4096 00020000:00200000:2.0:1789760740.957972:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.958056:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5719, 5719] 00000008:00010000:2.0:1789760740.958075:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.958081:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760740.958093:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23425024, 23429120) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.958107:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23425024=0x1657000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.958157:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.958159:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23429120 00000020:00010000:2.0:1789760740.958209:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.958214:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.958217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.958221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.958225:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.958232:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.958238:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.958253:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760740.958363:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 23425024<>23429120 00000100:00100000:3.0:1789760740.958427:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495812352:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.015998:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973036 gen 2 00000008:00010000:3.0:1789760741.016063:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760741.016071:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.016081:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.016106:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760741.016138:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495812352:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.016150:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5719, 5719] 00000080:00200000:2.0:1789760741.016216:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.016224:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.016240:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23429120 i_blocks 47111 00000020:00200000:2.0:1789760741.016244:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.016254:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.016288:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23429120, count: 4096 00000080:00200000:2.0:1789760741.016295:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23429120, bytes: 4096 00000080:00200000:2.0:1789760741.016305:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.016315:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.016317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23429120, 23433216) 0 0 00000080:00200000:2.0:1789760741.016322:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.016351:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5720, 5720] 00020000:00200000:2.0:1789760741.016360:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23429120 chunk: [23429120, 23433216) 23433216, 4096 00020000:00200000:2.0:1789760741.016365:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.016392:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5720, 5720] 00000008:00010000:2.0:1789760741.016409:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.016415:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.016427:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23429120, 23433216) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.016443:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23429120=0x1658000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.016498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.016502:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23433216 00000020:00010000:2.0:1789760741.016510:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.016514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.016517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.016520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.016523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.016529:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.016536:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.016551:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760741.016910:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 23429120<>23433216 00000100:00100000:3.0:1789760741.017149:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812480:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.054864:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973037 gen 2 00000008:00010000:3.0:1789760741.054914:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760741.054923:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.055131:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.055160:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760741.055224:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812480:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.055668:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5720, 5720] 00000080:00200000:2.0:1789760741.055686:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.055692:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.055718:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23433216 i_blocks 47111 00000020:00200000:2.0:1789760741.055723:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.055732:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.055764:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23433216, count: 4096 00000080:00200000:2.0:1789760741.055771:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23433216, bytes: 4096 00000080:00200000:2.0:1789760741.055791:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.055800:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.055803:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23433216, 23437312) 0 0 00000080:00200000:2.0:1789760741.055808:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.055825:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5721, 5721] 00020000:00200000:2.0:1789760741.055833:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23433216 chunk: [23433216, 23437312) 23437312, 4096 00020000:00200000:2.0:1789760741.055839:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.055859:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5721, 5721] 00000008:00010000:2.0:1789760741.055876:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.055882:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.055892:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23433216, 23437312) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.055905:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23433216=0x1659000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.055965:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.055968:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23437312 00000020:00010000:2.0:1789760741.055976:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.055980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.055984:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.055987:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.055989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.055997:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.056004:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.056020:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760741.056151:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 23433216<>23437312 00000100:00100000:3.0:1789760741.056519:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760741.056541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812608:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760741.107987:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760741.108002:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.108014:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.108043:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760741.108072:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812608:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760741.108472:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5721, 5721] 00000080:00200000:1.0:1789760741.108490:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760741.108496:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760741.108515:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23437312 i_blocks 47111 00000020:00200000:1.0:1789760741.108519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760741.108528:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760741.108560:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23437312, count: 4096 00000080:00200000:1.0:1789760741.108567:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23437312, bytes: 4096 00000080:00200000:1.0:1789760741.108577:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760741.108585:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760741.108588:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23437312, 23441408) 0 0 00000080:00200000:1.0:1789760741.108593:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760741.108607:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5722, 5722] 00020000:00200000:1.0:1789760741.108623:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23437312 chunk: [23437312, 23441408) 23441408, 4096 00020000:00200000:1.0:1789760741.108628:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760741.108646:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5722, 5722] 00000008:00010000:1.0:1789760741.108663:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760741.108668:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760741.109427:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23437312, 23441408) append 0 append lockpos 0 00000080:00200000:1.0:1789760741.109448:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23437312=0x165a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760741.109525:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760741.109527:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23441408 00000020:00010000:1.0:1789760741.109536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760741.109539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760741.109543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760741.109546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760741.109549:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760741.109558:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760741.109566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760741.109589:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760741.109737:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23437312<>23441408 00000100:00100000:1.0:1789760741.152500:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973039 gen 2 00000008:00010000:1.0:1789760741.152562:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760741.152570:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.152580:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.152692:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000080:00200000:2.0:1789760741.152693:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5722, 5722] 00000080:00200000:2.0:1789760741.152713:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.152720:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:1.0:1789760741.152724:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495812736:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.152739:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23441408 i_blocks 47111 00000020:00200000:2.0:1789760741.152744:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.152755:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.152791:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23441408, count: 4096 00000080:00200000:2.0:1789760741.152798:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23441408, bytes: 4096 00000080:00200000:2.0:1789760741.152808:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.152818:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.152821:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23441408, 23445504) 0 0 00000080:00200000:2.0:1789760741.152825:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.152841:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5723, 5723] 00020000:00200000:2.0:1789760741.152849:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23441408 chunk: [23441408, 23445504) 23445504, 4096 00020000:00200000:2.0:1789760741.152854:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.152873:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5723, 5723] 00000008:00010000:2.0:1789760741.152889:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.152895:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.152907:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23441408, 23445504) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.152922:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23441408=0x165b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.152978:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.152981:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23445504 00000020:00010000:2.0:1789760741.152990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.152994:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.152997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.153001:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.153004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.153011:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.153018:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.153033:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760741.153153:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 23441408<>23445504 00000100:00100000:3.0:1789760741.153249:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812864:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.183882:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973040 gen 2 00000008:00010000:3.0:1789760741.183939:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760741.183948:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.183959:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.183982:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760741.184008:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812864:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.184146:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5723, 5723] 00000080:00200000:2.0:1789760741.184163:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.184198:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.184215:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23445504 i_blocks 47111 00000020:00200000:2.0:1789760741.184219:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.184228:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.184264:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23445504, count: 4096 00000080:00200000:2.0:1789760741.184271:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23445504, bytes: 4096 00000080:00200000:2.0:1789760741.184279:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.184289:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.184291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23445504, 23449600) 0 0 00000080:00200000:2.0:1789760741.184295:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.184311:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5724, 5724] 00020000:00200000:2.0:1789760741.184319:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23445504 chunk: [23445504, 23449600) 23449600, 4096 00020000:00200000:2.0:1789760741.184324:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.184341:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5724, 5724] 00000008:00010000:2.0:1789760741.184357:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.184362:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.184372:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23445504, 23449600) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.184385:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23445504=0x165c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.184433:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.184435:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23449600 00000020:00010000:2.0:1789760741.184442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.184445:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.184448:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.184452:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.184454:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.184460:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.184467:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.184480:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760741.184599:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 23445504<>23449600 00000100:00100000:3.0:1789760741.184703:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760741.184719:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.232884:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973041 gen 2 00000008:00010000:3.0:1789760741.232934:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760741.232943:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.232954:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.232977:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760741.233020:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495812992:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.233069:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5724, 5724] 00000080:00200000:2.0:1789760741.233086:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.233092:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.233108:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23449600 i_blocks 47111 00000020:00200000:2.0:1789760741.233112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.233122:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.233156:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23449600, count: 4096 00000080:00200000:2.0:1789760741.233164:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23449600, bytes: 4096 00000080:00200000:2.0:1789760741.233202:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.233212:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.233215:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23449600, 23453696) 0 0 00000080:00200000:2.0:1789760741.233220:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.233242:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5725, 5725] 00020000:00200000:2.0:1789760741.233250:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23449600 chunk: [23449600, 23453696) 23453696, 4096 00020000:00200000:2.0:1789760741.233255:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.233275:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5725, 5725] 00000008:00010000:2.0:1789760741.233292:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.233298:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.233309:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23449600, 23453696) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.233322:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23449600=0x165d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.233382:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.233385:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23453696 00000020:00010000:2.0:1789760741.233394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.233398:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.233401:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.233404:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.233406:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.233414:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.233421:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.233436:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760741.233559:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 23449600<>23453696 00000100:00100000:3.0:1789760741.233717:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.302531:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973042 gen 2 00000008:00010000:3.0:1789760741.302584:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760741.302593:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.302617:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.302701:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760741.302731:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813120:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.303571:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5725, 5725] 00000080:00200000:2.0:1789760741.303591:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.303599:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.303624:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23453696 i_blocks 47111 00000020:00200000:2.0:1789760741.303629:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.303638:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.303673:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23453696, count: 4096 00000080:00200000:2.0:1789760741.303680:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23453696, bytes: 4096 00000080:00200000:2.0:1789760741.303693:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.303702:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.303704:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23453696, 23457792) 0 0 00000080:00200000:2.0:1789760741.303709:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.303725:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5726, 5726] 00020000:00200000:2.0:1789760741.303734:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23453696 chunk: [23453696, 23457792) 23457792, 4096 00020000:00200000:2.0:1789760741.303738:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.303758:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5726, 5726] 00000008:00010000:2.0:1789760741.303777:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.303784:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.303795:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23453696, 23457792) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.303809:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23453696=0x165e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.303871:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.303874:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23457792 00000020:00010000:2.0:1789760741.303882:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.303886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.303889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.303892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.303895:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.303903:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.303909:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.305130:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760741.305361:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 23453696<>23457792 00000100:00100000:2.0:1789760741.305419:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760741.305442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813248:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760741.349899:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973043 gen 2 00000008:00010000:2.0:1789760741.349946:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760741.349953:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760741.349964:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760741.350211:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760741.350240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813248:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760741.350335:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5726, 5726] 00000080:00200000:3.0:1789760741.350352:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760741.350358:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760741.350373:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23457792 i_blocks 47111 00000020:00200000:3.0:1789760741.350377:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760741.350386:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760741.350418:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23457792, count: 4096 00000080:00200000:3.0:1789760741.350426:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23457792, bytes: 4096 00000080:00200000:3.0:1789760741.350437:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760741.350447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760741.350449:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23457792, 23461888) 0 0 00000080:00200000:3.0:1789760741.350453:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760741.350469:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5727, 5727] 00020000:00200000:3.0:1789760741.350476:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23457792 chunk: [23457792, 23461888) 23461888, 4096 00020000:00200000:3.0:1789760741.350481:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760741.350501:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5727, 5727] 00000008:00010000:3.0:1789760741.350518:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760741.350524:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760741.350535:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23457792, 23461888) append 0 append lockpos 0 00000080:00200000:3.0:1789760741.350549:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23457792=0x165f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760741.350603:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760741.350606:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23461888 00000020:00010000:3.0:1789760741.350628:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760741.350632:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760741.350636:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760741.350640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760741.350643:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760741.350650:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760741.350657:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760741.350671:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760741.350793:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 23457792<>23461888 00000100:00100000:2.0:1789760741.350859:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813376:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760741.382544:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973044 gen 2 00000008:00010000:2.0:1789760741.382595:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760741.382620:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760741.382632:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760741.382654:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760741.382682:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813376:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760741.382850:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5727, 5727] 00000080:00200000:1.0:1789760741.382869:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760741.382875:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760741.382892:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23461888 i_blocks 47111 00000020:00200000:1.0:1789760741.382895:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760741.382905:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760741.382953:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23461888, count: 4096 00000080:00200000:1.0:1789760741.382961:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23461888, bytes: 4096 00000080:00200000:1.0:1789760741.382971:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760741.382979:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760741.382983:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23461888, 23465984) 0 0 00000080:00200000:1.0:1789760741.382987:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760741.383003:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5728, 5728] 00020000:00200000:1.0:1789760741.383012:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23461888 chunk: [23461888, 23465984) 23465984, 4096 00020000:00200000:1.0:1789760741.383016:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760741.383034:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5728, 5728] 00000008:00010000:1.0:1789760741.383050:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760741.383056:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760741.383067:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23461888, 23465984) append 0 append lockpos 0 00000080:00200000:1.0:1789760741.383081:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23461888=0x1660000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760741.383137:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760741.383139:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23465984 00000020:00010000:1.0:1789760741.383148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760741.383152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760741.383155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760741.383159:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760741.383162:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760741.383213:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760741.383221:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760741.383237:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760741.383362:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23461888<>23465984 00000100:00100000:1.0:1789760741.383397:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760741.383413:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495813504:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.420155:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973045 gen 2 00000008:00010000:1.0:1789760741.420241:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760741.420250:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.420262:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.420288:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760741.420318:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495813504:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.420427:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5728, 5728] 00000080:00200000:2.0:1789760741.420444:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.420450:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.420465:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23465984 i_blocks 47111 00000020:00200000:2.0:1789760741.420470:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.420479:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.420513:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23465984, count: 4096 00000080:00200000:2.0:1789760741.420521:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23465984, bytes: 4096 00000080:00200000:2.0:1789760741.420532:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.420540:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.420542:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23465984, 23470080) 0 0 00000080:00200000:2.0:1789760741.420547:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.420565:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5729, 5729] 00020000:00200000:2.0:1789760741.420573:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23465984 chunk: [23465984, 23470080) 23470080, 4096 00020000:00200000:2.0:1789760741.420578:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.420617:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5729, 5729] 00000008:00010000:2.0:1789760741.420636:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.420642:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.420654:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23465984, 23470080) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.420668:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23465984=0x1661000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.420723:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.420726:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23470080 00000020:00010000:2.0:1789760741.420735:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.420738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.420742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.420745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.420748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.420755:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.420761:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.420778:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760741.420901:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 23465984<>23470080 00000100:00100000:3.0:1789760741.421084:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495813632:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.456446:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973046 gen 2 00000008:00010000:3.0:1789760741.456495:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760741.456504:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.456515:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.456541:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760741.456572:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495813632:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.457096:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5729, 5729] 00000080:00200000:2.0:1789760741.457114:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.457120:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.457136:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23470080 i_blocks 47111 00000020:00200000:2.0:1789760741.457140:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.457149:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.457215:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23470080, count: 4096 00000080:00200000:2.0:1789760741.457223:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23470080, bytes: 4096 00000080:00200000:2.0:1789760741.457232:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.457241:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.457243:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23470080, 23474176) 0 0 00000080:00200000:2.0:1789760741.457247:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.457264:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5730, 5730] 00020000:00200000:2.0:1789760741.457273:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23470080 chunk: [23470080, 23474176) 23474176, 4096 00020000:00200000:2.0:1789760741.457277:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.457298:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5730, 5730] 00000008:00010000:2.0:1789760741.457316:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.457322:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.457334:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23470080, 23474176) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.457349:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23470080=0x1662000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.457407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.457410:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23474176 00000020:00010000:2.0:1789760741.457418:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.457422:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.457425:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.457428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.457431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.457439:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.457445:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.457463:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760741.457585:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 23470080<>23474176 00000100:00100000:3.0:1789760741.457727:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.484863:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973047 gen 2 00000008:00010000:3.0:1789760741.484914:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760741.484923:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.484942:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.484966:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760741.484996:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813760:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.485150:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5730, 5730] 00000080:00200000:2.0:1789760741.485166:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.485204:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.485220:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23474176 i_blocks 47111 00000020:00200000:2.0:1789760741.485224:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.485233:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.485263:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23474176, count: 4096 00000080:00200000:2.0:1789760741.485269:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23474176, bytes: 4096 00000080:00200000:2.0:1789760741.485279:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.485288:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.485290:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23474176, 23478272) 0 0 00000080:00200000:2.0:1789760741.485294:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.485309:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5731, 5731] 00020000:00200000:2.0:1789760741.485316:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23474176 chunk: [23474176, 23478272) 23478272, 4096 00020000:00200000:2.0:1789760741.485320:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.485338:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5731, 5731] 00000008:00010000:2.0:1789760741.485355:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.485361:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.485371:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23474176, 23478272) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.485385:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23474176=0x1663000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.485436:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.485439:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23478272 00000020:00010000:2.0:1789760741.485446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.485450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.485453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.485457:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.485460:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.485467:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.485473:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.485489:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760741.485620:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23474176<>23478272 00000100:00100000:3.0:1789760741.485709:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760741.485725:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813888:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.517480:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973048 gen 2 00000008:00010000:3.0:1789760741.517532:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760741.517541:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.517553:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.517648:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760741.517678:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495813888:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.517811:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5731, 5731] 00000080:00200000:2.0:1789760741.517830:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.517837:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.517853:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23478272 i_blocks 47111 00000020:00200000:2.0:1789760741.517858:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.517867:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.517901:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23478272, count: 4096 00000080:00200000:2.0:1789760741.517909:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23478272, bytes: 4096 00000080:00200000:2.0:1789760741.517920:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.517990:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.517993:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23478272, 23482368) 0 0 00000080:00200000:2.0:1789760741.517998:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.518014:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5732, 5732] 00020000:00200000:2.0:1789760741.518023:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23478272 chunk: [23478272, 23482368) 23482368, 4096 00020000:00200000:2.0:1789760741.518027:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.518048:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5732, 5732] 00000008:00010000:2.0:1789760741.518068:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.518074:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.518085:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23478272, 23482368) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.518098:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23478272=0x1664000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.518157:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.518160:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23482368 00000020:00010000:2.0:1789760741.518250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.518254:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.518257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.518261:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.518264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.518271:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.518279:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.518297:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760741.518412:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 23478272<>23482368 00000100:00100000:3.0:1789760741.518475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495814016:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.552136:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973049 gen 2 00000008:00010000:3.0:1789760741.552220:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760741.552229:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.552241:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.552264:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760741.552290:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495814016:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760741.552420:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5732, 5732] 00000080:00200000:2.0:1789760741.552437:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.552443:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.552459:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23482368 i_blocks 47111 00000020:00200000:2.0:1789760741.552464:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.552473:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.552504:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23482368, count: 4096 00000080:00200000:2.0:1789760741.552511:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23482368, bytes: 4096 00000080:00200000:2.0:1789760741.552521:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.552529:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.552531:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23482368, 23486464) 0 0 00000080:00200000:2.0:1789760741.552535:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.552553:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5733, 5733] 00020000:00200000:2.0:1789760741.552561:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23482368 chunk: [23482368, 23486464) 23486464, 4096 00020000:00200000:2.0:1789760741.552565:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.552584:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5733, 5733] 00000008:00010000:2.0:1789760741.552640:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.552646:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760741.552658:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23482368, 23486464) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.552672:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23482368=0x1665000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.552727:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.552730:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23486464 00000020:00010000:2.0:1789760741.552738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.552742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.552745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.552748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.552751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.552759:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.552766:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.552781:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760741.552900:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 23482368<>23486464 00000100:00100000:3.0:1789760741.552966:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495814144:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760741.590558:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973050 gen 2 00000008:00010000:3.0:1789760741.590622:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760741.590630:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.590641:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.590662:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760741.590699:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495814144:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.591575:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495814272:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.646694:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973051 gen 2 00000008:00010000:1.0:1789760741.646815:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760741.646825:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.646836:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.646857:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760741.646886:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495814272:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.647523:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495814400:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.684594:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973052 gen 2 00000008:00010000:1.0:1789760741.684659:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760741.684668:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.684678:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.684700:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760741.685818:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495814400:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.716285:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973053 gen 2 00000008:00010000:1.0:1789760741.716335:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760741.716343:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.716355:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.716379:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760741.716407:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495814528:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.717555:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760741.717577:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495814656:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.750356:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973054 gen 2 00000008:00010000:1.0:1789760741.750402:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760741.750410:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.750421:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.750442:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760741.750470:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495814656:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.751385:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495814784:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.780758:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973055 gen 2 00000008:00010000:1.0:1789760741.780812:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760741.780821:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.780833:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.780855:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760741.780884:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495814784:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.782152:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495814912:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.828752:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973056 gen 2 00000008:00010000:1.0:1789760741.828799:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760741.828807:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.828818:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.828840:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760741.828874:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495814912:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.829681:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815040:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760741.837571:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760741.855527:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973057 gen 2 00000008:00010000:1.0:1789760741.855724:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760741.855732:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.855743:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.855792:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760741.855817:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815040:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.856479:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495815168:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.880650:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973058 gen 2 00000008:00010000:1.0:1789760741.880697:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760741.880704:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.880715:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.880734:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760741.880761:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495815168:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.881423:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815296:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.906680:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973059 gen 2 00000008:00010000:1.0:1789760741.906733:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760741.906741:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.906752:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.906826:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760741.906866:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815296:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.907774:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495815424:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.943751:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973060 gen 2 00000008:00010000:1.0:1789760741.943807:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760741.943816:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.943828:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.943856:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760741.943889:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495815424:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.945111:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815552:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.971257:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973061 gen 2 00000008:00010000:1.0:1789760741.971309:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760741.971318:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.971329:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.971351:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760741.971381:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815552:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760741.972810:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760741.972831:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815680:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.009769:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973062 gen 2 00000008:00010000:1.0:1789760742.009815:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760742.009822:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.009830:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.009845:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760742.009880:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815680:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.010595:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815808:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.053861:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973063 gen 2 00000008:00010000:1.0:1789760742.053911:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760742.053920:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.053939:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.053962:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760742.053993:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815808:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.055402:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760742.055428:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815936:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.099895:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973064 gen 2 00000008:00010000:1.0:1789760742.099944:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760742.099954:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.099964:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.100050:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760742.100082:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495815936:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.100624:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5747, 5747] 00000080:00200000:2.0:1789760742.100650:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.100656:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.100701:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23543808 i_blocks 47111 00000020:00200000:2.0:1789760742.100705:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.100715:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.100750:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23543808, count: 4096 00000080:00200000:2.0:1789760742.100758:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23543808, bytes: 4096 00000080:00200000:2.0:1789760742.100769:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.100777:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.100779:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23543808, 23547904) 0 0 00000080:00200000:2.0:1789760742.100783:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.100800:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5748, 5748] 00020000:00200000:2.0:1789760742.100807:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23543808 chunk: [23543808, 23547904) 23547904, 4096 00020000:00200000:2.0:1789760742.100812:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.100832:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5748, 5748] 00000008:00010000:2.0:1789760742.100850:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.100856:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.100868:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23543808, 23547904) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.100883:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23543808=0x1674000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.100934:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.100938:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23547904 00000020:00010000:2.0:1789760742.100946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.100949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.100953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.100956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.100959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.100966:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.100973:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.101120:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760742.101347:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23543808<>23547904 00000100:00100000:3.0:1789760742.101438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495816064:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760742.159853:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760742.159870:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.159883:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.159909:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760742.159939:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495816064:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.160191:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5748, 5748] 00000080:00200000:2.0:1789760742.160210:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.160218:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.160236:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23547904 i_blocks 47111 00000020:00200000:2.0:1789760742.160240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.160250:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.160284:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23547904, count: 4096 00000080:00200000:2.0:1789760742.160292:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23547904, bytes: 4096 00000080:00200000:2.0:1789760742.160303:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.160312:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.160315:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23547904, 23552000) 0 0 00000080:00200000:2.0:1789760742.160319:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.160337:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5749, 5749] 00020000:00200000:2.0:1789760742.160345:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23547904 chunk: [23547904, 23552000) 23552000, 4096 00020000:00200000:2.0:1789760742.160351:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.160371:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5749, 5749] 00000008:00010000:2.0:1789760742.160390:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.160396:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.160409:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23547904, 23552000) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.160423:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23547904=0x1675000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.160479:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.160482:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23552000 00000020:00010000:2.0:1789760742.160491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.160494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.160498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.160501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.160504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.160511:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.160518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.160533:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760742.160666:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 23547904<>23552000 00000100:00100000:3.0:1789760742.161209:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495816192:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.192632:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973065 gen 2 00000100:00100000:3.0:1789760742.193063:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205267100 x1876699495816192/t4294973066(4294973066) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760758 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760742.193117:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760742.193124:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.193134:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.193164:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760742.193223:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495816192:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.193572:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5749, 5749] 00000080:00200000:2.0:1789760742.193609:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.193616:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.193633:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23552000 i_blocks 47111 00000020:00200000:2.0:1789760742.193637:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.193647:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.193680:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23552000, count: 4096 00000080:00200000:2.0:1789760742.193687:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23552000, bytes: 4096 00000080:00200000:2.0:1789760742.193701:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.193710:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.193712:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23552000, 23556096) 0 0 00000080:00200000:2.0:1789760742.193716:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.193733:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5750, 5750] 00020000:00200000:2.0:1789760742.193742:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23552000 chunk: [23552000, 23556096) 23556096, 4096 00020000:00200000:2.0:1789760742.193746:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.193767:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5750, 5750] 00000008:00010000:2.0:1789760742.193785:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.193791:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.193803:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23552000, 23556096) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.193817:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23552000=0x1676000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.193889:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.193892:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23556096 00000020:00010000:2.0:1789760742.193900:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.193909:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.193912:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.193915:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.193918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.193926:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.193933:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.193951:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760742.194080:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 23552000<>23556096 00000100:00100000:3.0:1789760742.194309:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495816320:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.242806:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973067 gen 2 00000008:00010000:3.0:1789760742.242870:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760742.242879:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.242889:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.242916:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760742.242957:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495816320:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.243249:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5750, 5750] 00000080:00200000:2.0:1789760742.243268:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.243274:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.243292:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23556096 i_blocks 47111 00000020:00200000:2.0:1789760742.243297:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.243308:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.243343:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23556096, count: 4096 00000080:00200000:2.0:1789760742.243350:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23556096, bytes: 4096 00000080:00200000:2.0:1789760742.243362:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.243371:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.243374:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23556096, 23560192) 0 0 00000080:00200000:2.0:1789760742.243378:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.243396:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5751, 5751] 00020000:00200000:2.0:1789760742.243404:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23556096 chunk: [23556096, 23560192) 23560192, 4096 00020000:00200000:2.0:1789760742.243409:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.243428:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5751, 5751] 00000008:00010000:2.0:1789760742.243445:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.243451:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.243464:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23556096, 23560192) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.243478:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23556096=0x1677000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.243538:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.243541:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23560192 00000020:00010000:2.0:1789760742.243550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.243554:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.243557:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.243561:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.243564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.243572:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.243592:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.243618:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760742.243747:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 23556096<>23560192 00000100:00100000:3.0:1789760742.244030:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495816448:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760742.275235:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760742.275252:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.275262:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.275288:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760742.275317:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495816448:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.275458:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5751, 5751] 00000080:00200000:2.0:1789760742.275476:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.275483:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.275498:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23560192 i_blocks 47111 00000020:00200000:2.0:1789760742.275502:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.275512:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.275548:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23560192, count: 4096 00000080:00200000:2.0:1789760742.275555:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23560192, bytes: 4096 00000080:00200000:2.0:1789760742.275565:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.275573:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.275576:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23560192, 23564288) 0 0 00000080:00200000:2.0:1789760742.275591:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.275617:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5752, 5752] 00020000:00200000:2.0:1789760742.275627:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23560192 chunk: [23560192, 23564288) 23564288, 4096 00020000:00200000:2.0:1789760742.275632:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.275653:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5752, 5752] 00000008:00010000:2.0:1789760742.275670:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.275676:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.275688:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23560192, 23564288) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.275701:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23560192=0x1678000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.275761:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.275764:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23564288 00000020:00010000:2.0:1789760742.275772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.275776:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.275779:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.275783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.275786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.275794:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.275801:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.275816:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760742.276104:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 23560192<>23564288 00000100:00100000:3.0:1789760742.276235:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495816576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.308381:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973068 gen 2 00000100:00100000:3.0:1789760742.308424:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205267800 x1876699495816576/t4294973069(4294973069) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760758 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760742.308477:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760742.308484:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.308495:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.308521:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760742.308547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495816576:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.308674:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5752, 5752] 00000080:00200000:2.0:1789760742.308690:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.308697:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.308713:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23564288 i_blocks 47111 00000020:00200000:2.0:1789760742.308717:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.308726:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.308764:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23564288, count: 4096 00000080:00200000:2.0:1789760742.308783:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23564288, bytes: 4096 00000080:00200000:2.0:1789760742.308794:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.308804:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.308807:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23564288, 23568384) 0 0 00000080:00200000:2.0:1789760742.308812:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.308843:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5753, 5753] 00020000:00200000:2.0:1789760742.308853:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23564288 chunk: [23564288, 23568384) 23568384, 4096 00020000:00200000:2.0:1789760742.308858:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.308880:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5753, 5753] 00000008:00010000:2.0:1789760742.308898:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.308904:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.308915:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23564288, 23568384) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.308929:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23564288=0x1679000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.308991:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.308994:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23568384 00000020:00010000:2.0:1789760742.309003:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.309007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.309010:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.309014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.309018:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.309026:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.309034:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.309052:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760742.309251:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 23564288<>23568384 00000100:00100000:3.0:1789760742.309489:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495816704:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.352819:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973070 gen 2 00000008:00010000:3.0:1789760742.352885:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760742.352893:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.352904:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.352927:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760742.352956:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495816704:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.356751:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5753, 5753] 00000080:00200000:2.0:1789760742.356779:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.356787:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.356806:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23568384 i_blocks 47111 00000020:00200000:2.0:1789760742.356812:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.356829:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.356866:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23568384, count: 4096 00000080:00200000:2.0:1789760742.356874:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23568384, bytes: 4096 00000080:00200000:2.0:1789760742.356889:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.356898:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.356901:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23568384, 23572480) 0 0 00000080:00200000:2.0:1789760742.356905:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.356922:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5754, 5754] 00020000:00200000:2.0:1789760742.356932:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23568384 chunk: [23568384, 23572480) 23572480, 4096 00020000:00200000:2.0:1789760742.356936:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.356959:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5754, 5754] 00000008:00010000:2.0:1789760742.356979:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.356984:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.356996:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23568384, 23572480) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.357013:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23568384=0x167a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.357083:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.357086:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23572480 00000020:00010000:2.0:1789760742.357094:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.357098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.357101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.357104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.357107:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.357116:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.357122:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.357144:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760742.357327:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 23568384<>23572480 00000100:00100000:3.0:1789760742.357615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495816832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.387091:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973071 gen 2 00000008:00010000:3.0:1789760742.387139:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760742.387147:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.387158:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.387225:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760742.387253:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495816832:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.387402:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5754, 5754] 00000080:00200000:2.0:1789760742.387420:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.387427:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.387445:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23572480 i_blocks 47111 00000020:00200000:2.0:1789760742.387449:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.387459:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.387495:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23572480, count: 4096 00000080:00200000:2.0:1789760742.387503:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23572480, bytes: 4096 00000080:00200000:2.0:1789760742.387514:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.387524:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.387526:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23572480, 23576576) 0 0 00000080:00200000:2.0:1789760742.387531:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.387550:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5755, 5755] 00020000:00200000:2.0:1789760742.387559:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23572480 chunk: [23572480, 23576576) 23576576, 4096 00020000:00200000:2.0:1789760742.387565:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.387598:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5755, 5755] 00000008:00010000:2.0:1789760742.387627:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.387633:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.387645:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23572480, 23576576) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.387659:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23572480=0x167b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.387716:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.387719:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23576576 00000020:00010000:2.0:1789760742.387727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.387731:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.387734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.387738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.387741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.387748:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.387756:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.387773:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760742.387904:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 23572480<>23576576 00000100:00100000:3.0:1789760742.388085:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495816960:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.420146:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973072 gen 2 00000008:00010000:3.0:1789760742.420227:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760742.420235:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.420246:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.420270:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760742.420299:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495816960:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.420554:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5755, 5755] 00000080:00200000:2.0:1789760742.420572:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.420587:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.420611:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23576576 i_blocks 47111 00000020:00200000:2.0:1789760742.420615:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.420625:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.420658:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23576576, count: 4096 00000080:00200000:2.0:1789760742.420665:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23576576, bytes: 4096 00000080:00200000:2.0:1789760742.420674:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.420683:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.420685:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23576576, 23580672) 0 0 00000080:00200000:2.0:1789760742.420689:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.420705:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5756, 5756] 00020000:00200000:2.0:1789760742.420713:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23576576 chunk: [23576576, 23580672) 23580672, 4096 00020000:00200000:2.0:1789760742.420718:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.420739:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5756, 5756] 00000008:00010000:2.0:1789760742.420757:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.420763:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.420775:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23576576, 23580672) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.420789:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23576576=0x167c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.420859:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.420862:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23580672 00000020:00010000:2.0:1789760742.420871:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.420874:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.420878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.420881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.420884:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.420891:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.420898:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.420914:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760742.421228:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 23576576<>23580672 00000100:00100000:3.0:1789760742.421337:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760742.421357:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495817088:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760742.451241:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760742.451270:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.451284:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.451304:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760742.451338:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495817088:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.451814:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5756, 5756] 00000080:00200000:2.0:1789760742.451832:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.451839:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.451865:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23580672 i_blocks 47111 00000020:00200000:2.0:1789760742.451869:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.451879:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.451914:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23580672, count: 4096 00000080:00200000:2.0:1789760742.452205:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23580672, bytes: 4096 00000080:00200000:2.0:1789760742.452219:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.452230:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.452232:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23580672, 23584768) 0 0 00000080:00200000:2.0:1789760742.452237:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.452255:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5757, 5757] 00020000:00200000:2.0:1789760742.452264:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23580672 chunk: [23580672, 23584768) 23584768, 4096 00020000:00200000:2.0:1789760742.452268:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.452294:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5757, 5757] 00000008:00010000:2.0:1789760742.452318:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.452325:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.452340:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23580672, 23584768) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.452357:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23580672=0x167d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.452425:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.452428:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23584768 00000020:00010000:2.0:1789760742.452436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.452439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.452442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.452446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.452449:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.452456:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.452463:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.452482:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760742.452635:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 23580672<>23584768 00000100:00100000:3.0:1789760742.452714:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495817216:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.489090:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973073 gen 2 00000100:00100000:3.0:1789760742.489128:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495817216/t4294973074(4294973074) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760758 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760742.489214:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760742.489222:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.489232:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.489253:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760742.489274:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495817216:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.490221:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5757, 5757] 00000080:00200000:2.0:1789760742.490246:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.490253:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.490269:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23584768 i_blocks 47111 00000020:00200000:2.0:1789760742.490273:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.490284:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.490318:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23584768, count: 4096 00000080:00200000:2.0:1789760742.490326:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23584768, bytes: 4096 00000080:00200000:2.0:1789760742.490337:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.490347:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.490349:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23584768, 23588864) 0 0 00000080:00200000:2.0:1789760742.490353:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.490371:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5758, 5758] 00020000:00200000:2.0:1789760742.490380:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23584768 chunk: [23584768, 23588864) 23588864, 4096 00020000:00200000:2.0:1789760742.490385:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.490407:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5758, 5758] 00000008:00010000:2.0:1789760742.490430:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.490437:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.490449:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23584768, 23588864) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.490464:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23584768=0x167e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.490526:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.490529:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23588864 00000020:00010000:2.0:1789760742.490539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.490543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.490546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.490550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.490553:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.490562:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.490568:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.490585:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760742.490738:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 23584768<>23588864 00000100:00100000:3.0:1789760742.491213:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495817344:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.530524:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973075 gen 2 00000008:00010000:3.0:1789760742.530590:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760742.530612:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.532492:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.532541:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760742.532584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495817344:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.532918:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5758, 5758] 00000080:00200000:2.0:1789760742.532938:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.532944:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.532964:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23588864 i_blocks 47111 00000020:00200000:2.0:1789760742.532968:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.532978:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.533013:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23588864, count: 4096 00000080:00200000:2.0:1789760742.533021:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23588864, bytes: 4096 00000080:00200000:2.0:1789760742.533039:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.533048:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.533051:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23588864, 23592960) 0 0 00000080:00200000:2.0:1789760742.533056:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.533153:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5759, 5759] 00020000:00200000:2.0:1789760742.533162:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23588864 chunk: [23588864, 23592960) 23592960, 4096 00020000:00200000:2.0:1789760742.533195:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.533219:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5759, 5759] 00000008:00010000:2.0:1789760742.533240:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.533246:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.533258:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23588864, 23592960) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.533273:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23588864=0x167f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.533337:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.534912:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23592960 00000020:00010000:2.0:1789760742.534932:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.534937:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.534942:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.534947:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.534950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.534961:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.534971:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.535009:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760742.535214:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 23588864<>23592960 00000100:00100000:3.0:1789760742.535458:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760742.535488:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495817472:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760742.566892:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760742.566909:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.566931:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.566957:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760742.566989:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495817472:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.567226:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5759, 5759] 00000080:00200000:2.0:1789760742.567317:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.567325:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.567342:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23592960 i_blocks 47111 00000020:00200000:2.0:1789760742.567346:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.567355:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.567390:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23592960, count: 4096 00000080:00200000:2.0:1789760742.567397:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23592960, bytes: 4096 00000080:00200000:2.0:1789760742.567411:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.567421:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.567423:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23592960, 23597056) 0 0 00000080:00200000:2.0:1789760742.567428:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.567445:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5760, 5760] 00020000:00200000:2.0:1789760742.567453:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23592960 chunk: [23592960, 23597056) 23597056, 4096 00020000:00200000:2.0:1789760742.567458:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.567477:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5760, 5760] 00000008:00010000:2.0:1789760742.567495:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.567501:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.567513:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23592960, 23597056) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.567528:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23592960=0x1680000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.567585:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.567588:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23597056 00000020:00010000:2.0:1789760742.567597:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.567611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.567613:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.567617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.567620:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.567628:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.567635:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.567652:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760742.569328:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 23592960<>23597056 00000100:00100000:3.0:1789760742.569414:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495817600:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.604606:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973076 gen 2 00000100:00100000:3.0:1789760742.604649:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264000 x1876699495817600/t4294973077(4294973077) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760758 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760742.604701:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760742.604708:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.604719:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.604747:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760742.604774:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495817600:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760742.605040:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5760, 5760] 00000080:00200000:1.0:1789760742.605057:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.605064:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.605080:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23597056 i_blocks 47111 00000020:00200000:1.0:1789760742.605085:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.605095:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.605131:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23597056, count: 4096 00000080:00200000:1.0:1789760742.605138:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23597056, bytes: 4096 00000080:00200000:1.0:1789760742.605149:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.605159:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.605161:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23597056, 23601152) 0 0 00000080:00200000:1.0:1789760742.605166:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.605225:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5761, 5761] 00020000:00200000:1.0:1789760742.605235:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23597056 chunk: [23597056, 23601152) 23601152, 4096 00020000:00200000:1.0:1789760742.605240:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.605265:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5761, 5761] 00000008:00010000:1.0:1789760742.605286:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.605292:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760742.605306:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23597056, 23601152) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.605322:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23597056=0x1681000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.605383:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.605386:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23601152 00000020:00010000:1.0:1789760742.605397:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.605401:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.605404:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.605409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.605412:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.605420:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.605427:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.605446:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760742.605596:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23597056<>23601152 00000100:00100000:1.0:1789760742.605660:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760742.605679:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495817728:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.634522:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973078 gen 2 00000008:00010000:1.0:1789760742.634606:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760742.634616:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.634627:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.634649:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760742.634677:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495817728:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.667587:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973079 gen 2 00000008:00010000:1.0:1789760742.667647:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760742.667655:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.667667:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.667690:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760742.667719:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495817856:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.668666:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495817984:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.705522:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973080 gen 2 00000008:00010000:1.0:1789760742.705790:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760742.705799:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.705810:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.705831:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760742.705861:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495817984:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760742.750474:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760742.750491:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.750503:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.750581:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760742.750622:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495818112:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760742.751114:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5764, 5764] 00000080:00200000:3.0:1789760742.751133:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760742.751140:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760742.751156:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23613440 i_blocks 47111 00000020:00200000:3.0:1789760742.751160:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760742.751274:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.751472:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23613440, count: 4096 00000080:00200000:2.0:1789760742.751482:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23613440, bytes: 4096 00000080:00200000:2.0:1789760742.751495:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.751505:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.751507:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23613440, 23617536) 0 0 00000080:00200000:2.0:1789760742.751512:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.751531:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5765, 5765] 00020000:00200000:2.0:1789760742.751539:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23613440 chunk: [23613440, 23617536) 23617536, 4096 00020000:00200000:2.0:1789760742.751544:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.751608:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5765, 5765] 00000008:00010000:2.0:1789760742.751626:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.751632:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.751645:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23613440, 23617536) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.751661:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23613440=0x1685000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.751939:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.751943:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23617536 00000020:00010000:2.0:1789760742.751953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.751956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.751960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.751963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.751966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.751974:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.751981:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.752017:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760742.752161:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 23613440<>23617536 00000100:00100000:3.0:1789760742.752257:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495818240:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760742.788062:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973082 gen 2 00000008:00010000:2.0:1789760742.788124:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760742.788133:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760742.788144:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760742.790063:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760742.790110:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495818240:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760742.790680:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5765, 5765] 00000080:00200000:1.0:1789760742.790699:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.790705:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.790722:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23617536 i_blocks 47111 00000020:00200000:1.0:1789760742.790727:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.790739:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.791880:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23617536, count: 4096 00000080:00200000:1.0:1789760742.791895:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23617536, bytes: 4096 00000080:00200000:1.0:1789760742.791911:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.791988:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.791991:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23617536, 23621632) 0 0 00000080:00200000:1.0:1789760742.791997:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.792016:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5766, 5766] 00020000:00200000:1.0:1789760742.792025:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23617536 chunk: [23617536, 23621632) 23621632, 4096 00020000:00200000:1.0:1789760742.792030:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.792051:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5766, 5766] 00000008:00010000:1.0:1789760742.792073:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.792079:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760742.792092:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23617536, 23621632) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.792107:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23617536=0x1686000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.792307:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.792311:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23621632 00000020:00010000:1.0:1789760742.792319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.792323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.792327:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.792331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.792334:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.792345:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.792354:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.792381:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760742.792514:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23617536<>23621632 00000100:00100000:1.0:1789760742.796212:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760742.796249:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495818368:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.836777:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973083 gen 2 00000008:00010000:1.0:1789760742.836970:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760742.836979:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.836991:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.837017:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760742.837047:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495818368:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760742.837233:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5766, 5766] 00000080:00200000:2.0:1789760742.837311:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.837318:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.837333:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23621632 i_blocks 47111 00000020:00200000:2.0:1789760742.837337:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.837346:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.837382:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23621632, count: 4096 00000080:00200000:2.0:1789760742.837389:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23621632, bytes: 4096 00000080:00200000:2.0:1789760742.837399:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.837408:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.837410:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23621632, 23625728) 0 0 00000080:00200000:2.0:1789760742.837414:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.837429:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5767, 5767] 00020000:00200000:2.0:1789760742.837437:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23621632 chunk: [23621632, 23625728) 23625728, 4096 00020000:00200000:2.0:1789760742.837441:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.837459:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5767, 5767] 00000008:00010000:2.0:1789760742.837477:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.837483:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760742.837494:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23621632, 23625728) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.837508:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23621632=0x1687000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.837566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.837569:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23625728 00000020:00010000:2.0:1789760742.837578:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.837582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.837586:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.837589:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.837593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.837647:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.837654:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.837671:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760742.837799:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 23621632<>23625728 00000100:00100000:3.0:1789760742.837874:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495818496:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760742.887400:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973084 gen 2 00000008:00010000:3.0:1789760742.887453:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760742.887462:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.887475:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.887552:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760742.887593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495818496:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760742.888018:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5767, 5767] 00000080:00200000:1.0:1789760742.888035:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.888041:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.888056:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23625728 i_blocks 47111 00000020:00200000:1.0:1789760742.888059:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.888068:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.888104:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23625728, count: 4096 00000080:00200000:1.0:1789760742.888111:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23625728, bytes: 4096 00000080:00200000:1.0:1789760742.888121:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.888131:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.888133:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23625728, 23629824) 0 0 00000080:00200000:1.0:1789760742.888138:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.888155:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5768, 5768] 00020000:00200000:1.0:1789760742.888164:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23625728 chunk: [23625728, 23629824) 23629824, 4096 00020000:00200000:1.0:1789760742.888208:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.888229:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5768, 5768] 00000008:00010000:1.0:1789760742.888247:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.888253:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760742.888265:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23625728, 23629824) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.888281:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23625728=0x1688000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.888340:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.888344:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23629824 00000020:00010000:1.0:1789760742.888352:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.888356:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.888359:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.888363:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.888366:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.888374:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.888380:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.888399:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760742.888763:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23625728<>23629824 00000008:00010000:1.0:1789760742.924882:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760742.924896:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.924906:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.924920:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760742.924951:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495818624:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.973263:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973086 gen 2 00000008:00010000:1.0:1789760742.973326:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760742.973334:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.973344:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.973402:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760742.973431:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495818752:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760742.974369:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495818880:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760743.007116:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973087 gen 2 00000008:00010000:1.0:1789760743.007213:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760743.007223:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.007234:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.007256:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760743.007279:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495818880:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760743.053810:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973088 gen 2 00000008:00010000:1.0:1789760743.054016:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760743.054025:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.054036:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.054127:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760743.054156:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495819008:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.055000:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5771, 5771] 00000080:00200000:2.0:1789760743.055021:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.055027:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.055044:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23642112 i_blocks 47111 00000020:00200000:2.0:1789760743.055049:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.055062:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.055098:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23642112, count: 4096 00000080:00200000:2.0:1789760743.055105:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23642112, bytes: 4096 00000080:00200000:2.0:1789760743.055198:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.055208:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.055211:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23642112, 23646208) 0 0 00000080:00200000:2.0:1789760743.055216:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.055233:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5772, 5772] 00020000:00200000:2.0:1789760743.055241:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23642112 chunk: [23642112, 23646208) 23646208, 4096 00020000:00200000:2.0:1789760743.055245:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.055265:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5772, 5772] 00000008:00010000:2.0:1789760743.055283:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.055289:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.055301:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23642112, 23646208) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.055315:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23642112=0x168c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.055381:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.055384:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23646208 00000020:00010000:2.0:1789760743.055392:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.055396:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.055399:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.055402:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.055404:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.055412:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.055418:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.055441:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760743.055563:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23642112<>23646208 00000100:00100000:3.0:1789760743.055663:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495819136:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760743.106489:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760743.106506:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.106519:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.106737:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760743.106772:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495819136:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.106994:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5772, 5772] 00000080:00200000:2.0:1789760743.107014:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.107022:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.107041:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23646208 i_blocks 47111 00000020:00200000:2.0:1789760743.107045:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.107056:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.107098:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23646208, count: 4096 00000080:00200000:2.0:1789760743.107106:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23646208, bytes: 4096 00000080:00200000:2.0:1789760743.107123:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.107135:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.107138:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23646208, 23650304) 0 0 00000080:00200000:2.0:1789760743.107143:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.107163:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5773, 5773] 00020000:00200000:2.0:1789760743.107203:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23646208 chunk: [23646208, 23650304) 23650304, 4096 00020000:00200000:2.0:1789760743.107208:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.107230:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5773, 5773] 00000008:00010000:2.0:1789760743.107444:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.107451:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.107464:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23646208, 23650304) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.107480:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23646208=0x168d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.107541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.107544:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23650304 00000020:00010000:2.0:1789760743.107552:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.107556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.107559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.107563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.107566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.107651:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.107660:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.107682:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760743.107815:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 23646208<>23650304 00000100:00100000:3.0:1789760743.107904:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495819264:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760743.150557:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973090 gen 2 00000008:00010000:3.0:1789760743.150635:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760743.150644:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.150655:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.150738:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760743.150771:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495819264:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.155095:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5773, 5773] 00000080:00200000:2.0:1789760743.155137:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.155145:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.155194:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23650304 i_blocks 47111 00000020:00200000:2.0:1789760743.155200:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.155219:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.155262:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23650304, count: 4096 00000080:00200000:2.0:1789760743.155270:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23650304, bytes: 4096 00000080:00200000:2.0:1789760743.155286:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.155296:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.155299:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23650304, 23654400) 0 0 00000080:00200000:2.0:1789760743.155303:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.155330:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5774, 5774] 00020000:00200000:2.0:1789760743.155340:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23650304 chunk: [23650304, 23654400) 23654400, 4096 00020000:00200000:2.0:1789760743.155345:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.155368:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5774, 5774] 00000008:00010000:2.0:1789760743.155389:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.155395:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.155407:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23650304, 23654400) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.155421:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23650304=0x168e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.155795:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.155799:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23654400 00000020:00010000:2.0:1789760743.155809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.155813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.155816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.155819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.156268:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.156282:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.156290:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.156367:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760743.156500:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 23650304<>23654400 00000100:00100000:3.0:1789760743.156587:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495819392:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760743.204131:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973091 gen 2 00000008:00010000:3.0:1789760743.204426:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760743.204435:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.204448:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.204542:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760743.204582:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495819392:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.204746:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5774, 5774] 00000080:00200000:2.0:1789760743.204763:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.204770:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.204785:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23654400 i_blocks 47111 00000020:00200000:2.0:1789760743.204789:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.204799:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.204834:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23654400, count: 4096 00000080:00200000:2.0:1789760743.204841:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23654400, bytes: 4096 00000080:00200000:2.0:1789760743.204981:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.204992:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.204994:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23654400, 23658496) 0 0 00000080:00200000:2.0:1789760743.204998:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.205015:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5775, 5775] 00020000:00200000:2.0:1789760743.205023:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23654400 chunk: [23654400, 23658496) 23658496, 4096 00020000:00200000:2.0:1789760743.205027:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.205047:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5775, 5775] 00000008:00010000:2.0:1789760743.205066:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.205072:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.205082:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23654400, 23658496) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.205095:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23654400=0x168f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.205293:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.205298:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23658496 00000020:00010000:2.0:1789760743.205307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.205434:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.205438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.205441:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.205444:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.205453:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.205460:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.205481:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760743.205685:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23654400<>23658496 00000100:00100000:3.0:1789760743.205762:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495819520:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760743.250746:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973092 gen 2 00000008:00010000:3.0:1789760743.250888:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760743.250898:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.250908:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.251022:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760743.251049:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495819520:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760743.251449:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5776, 5776] 00020000:00200000:0.0:1789760743.251456:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23658496 chunk: [23658496, 23662592) 23662592, 4096 00020000:00200000:0.0:1789760743.251459:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760743.251473:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5776, 5776] 00000008:00010000:0.0:1789760743.251485:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760743.251490:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760743.251498:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23658496, 23662592) append 0 append lockpos 0 00000080:00200000:0.0:1789760743.251508:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23658496=0x1690000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760743.251551:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760743.251553:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23662592 00000020:00010000:0.0:1789760743.251560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760743.251564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.251566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.251570:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760743.251572:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760743.251578:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760743.252528:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760743.252555:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760743.252686:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 23658496<>23662592 00000100:00100000:1.0:1789760743.252809:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.252829:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495819648:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760743.286920:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760743.286936:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.286949:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.286978:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760743.287016:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495819648:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760743.287347:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5776, 5776] 00000080:00200000:0.0:1789760743.287365:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760743.287371:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760743.287386:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23662592 i_blocks 47111 00000020:00200000:0.0:1789760743.287390:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760743.287399:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760743.287433:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23662592, count: 4096 00000080:00200000:0.0:1789760743.287440:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23662592, bytes: 4096 00000080:00200000:0.0:1789760743.287456:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760743.287465:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760743.287468:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23662592, 23666688) 0 0 00000080:00200000:0.0:1789760743.287473:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760743.287494:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5777, 5777] 00020000:00200000:0.0:1789760743.287503:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23662592 chunk: [23662592, 23666688) 23666688, 4096 00020000:00200000:0.0:1789760743.287508:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760743.287528:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5777, 5777] 00000008:00010000:0.0:1789760743.287546:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760743.287552:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760743.287564:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23662592, 23666688) append 0 append lockpos 0 00000080:00200000:0.0:1789760743.287579:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23662592=0x1691000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760743.287654:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760743.287657:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23666688 00000020:00010000:0.0:1789760743.287666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760743.287670:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.287673:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.287676:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760743.287680:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760743.287687:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760743.287694:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760743.287709:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760743.287837:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 23662592<>23666688 00000100:00100000:1.0:1789760743.287949:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495819776:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760743.307565:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495820032:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760743.307707:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760743.307714:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2359:1876699495819904:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760743.307754:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2359:1876699495820160:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760743.312475:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495820032:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760743.315809:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2359:1876699495820160:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760743.316215:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2359:1876699495819904:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760743.335330:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973094 gen 2 00000008:00010000:1.0:1789760743.335397:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760743.335405:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.335415:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.335670:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760743.335702:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495819776:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760743.335986:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5777, 5777] 00000080:00200000:0.0:1789760743.336005:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760743.336011:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760743.336027:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23666688 i_blocks 47111 00000020:00200000:0.0:1789760743.336030:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760743.336039:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760743.336076:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23666688, count: 4096 00000080:00200000:0.0:1789760743.336084:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23666688, bytes: 4096 00000080:00200000:0.0:1789760743.336094:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760743.336103:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760743.336105:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23666688, 23670784) 0 0 00000080:00200000:0.0:1789760743.336110:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760743.336130:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5778, 5778] 00020000:00200000:0.0:1789760743.336141:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23666688 chunk: [23666688, 23670784) 23670784, 4096 00020000:00200000:0.0:1789760743.336146:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760743.336268:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5778, 5778] 00000008:00010000:0.0:1789760743.336289:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760743.336295:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760743.336307:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23666688, 23670784) append 0 append lockpos 0 00000080:00200000:0.0:1789760743.336322:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23666688=0x1692000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760743.336383:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760743.336386:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23670784 00000020:00010000:0.0:1789760743.336396:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760743.336400:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.336403:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.336407:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760743.336410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760743.336418:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760743.336425:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760743.336444:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760743.337347:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 23666688<>23670784 00000100:00100000:1.0:1789760743.337518:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.337543:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820288:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760743.384295:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760743.384311:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.384323:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.384345:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760743.384378:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820288:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760743.384477:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5778, 5778] 00000080:00200000:0.0:1789760743.384494:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760743.384500:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760743.384516:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23670784 i_blocks 47111 00000020:00200000:0.0:1789760743.384520:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760743.384529:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760743.384565:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23670784, count: 4096 00000080:00200000:0.0:1789760743.384584:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23670784, bytes: 4096 00000080:00200000:0.0:1789760743.384602:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760743.384612:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760743.384615:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23670784, 23674880) 0 0 00000080:00200000:0.0:1789760743.384619:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760743.384636:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5779, 5779] 00020000:00200000:0.0:1789760743.384645:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23670784 chunk: [23670784, 23674880) 23674880, 4096 00020000:00200000:0.0:1789760743.384649:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760743.384669:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5779, 5779] 00000008:00010000:0.0:1789760743.384687:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760743.384694:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760743.384705:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23670784, 23674880) append 0 append lockpos 0 00000080:00200000:0.0:1789760743.384719:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23670784=0x1693000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760743.384777:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760743.384780:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23674880 00000020:00010000:0.0:1789760743.384790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760743.384794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.384797:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.384800:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760743.384803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760743.384810:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760743.384818:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760743.384836:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760743.385077:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 23670784<>23674880 00000100:00100000:1.0:1789760743.385196:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820416:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760743.430153:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973096 gen 2 00000008:00010000:1.0:1789760743.430249:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760743.430257:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.430267:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.430937:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760743.430972:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820416:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760743.431335:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5779, 5779] 00000080:00200000:0.0:1789760743.431352:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760743.431358:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760743.431374:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23674880 i_blocks 47111 00000020:00200000:0.0:1789760743.431379:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760743.431388:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760743.431423:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23674880, count: 4096 00000080:00200000:0.0:1789760743.431431:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23674880, bytes: 4096 00000080:00200000:0.0:1789760743.431444:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760743.431453:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760743.431455:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23674880, 23678976) 0 0 00000080:00200000:0.0:1789760743.431460:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760743.431477:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5780, 5780] 00020000:00200000:0.0:1789760743.431485:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23674880 chunk: [23674880, 23678976) 23678976, 4096 00020000:00200000:0.0:1789760743.431490:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760743.431510:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5780, 5780] 00000008:00010000:0.0:1789760743.431526:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760743.431531:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760743.431542:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23674880, 23678976) append 0 append lockpos 0 00000080:00200000:0.0:1789760743.431557:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23674880=0x1694000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760743.431624:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760743.431627:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23678976 00000020:00010000:0.0:1789760743.431635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760743.431639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.431642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.431645:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760743.431648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760743.431655:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760743.431661:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760743.431678:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760743.431805:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 23674880<>23678976 00000100:00100000:1.0:1789760743.431874:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495820544:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760743.484997:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973097 gen 2 00000008:00010000:1.0:1789760743.485048:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760743.485057:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.485069:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.485136:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760743.485167:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495820544:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760743.485525:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5780, 5780] 00000080:00200000:0.0:1789760743.485542:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760743.485550:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760743.485568:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23678976 i_blocks 47111 00000020:00200000:0.0:1789760743.485573:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760743.485583:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760743.485627:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23678976, count: 4096 00000080:00200000:0.0:1789760743.485634:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23678976, bytes: 4096 00000080:00200000:0.0:1789760743.485644:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760743.485652:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760743.485654:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23678976, 23683072) 0 0 00000080:00200000:0.0:1789760743.485659:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760743.485822:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5781, 5781] 00020000:00200000:0.0:1789760743.485835:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23678976 chunk: [23678976, 23683072) 23683072, 4096 00020000:00200000:0.0:1789760743.485840:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760743.485862:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5781, 5781] 00000008:00010000:0.0:1789760743.485882:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760743.485888:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760743.485901:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23678976, 23683072) append 0 append lockpos 0 00000080:00200000:0.0:1789760743.485918:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23678976=0x1695000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760743.485979:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760743.485982:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23683072 00000020:00010000:0.0:1789760743.485990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760743.485994:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.485996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.486001:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760743.486004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760743.486012:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760743.486019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760743.486037:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760743.486166:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 23678976<>23683072 00000100:00100000:1.0:1789760743.487037:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820672:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760743.539296:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760743.539329:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.539344:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.539412:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760743.539448:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820672:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760743.539890:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5781, 5781] 00000080:00200000:0.0:1789760743.539907:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760743.539914:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760743.539931:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23683072 i_blocks 47111 00000020:00200000:0.0:1789760743.539935:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760743.539945:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760743.539983:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23683072, count: 4096 00000080:00200000:0.0:1789760743.539991:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23683072, bytes: 4096 00000080:00200000:0.0:1789760743.540004:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760743.540014:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760743.540017:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23683072, 23687168) 0 0 00000080:00200000:0.0:1789760743.540022:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760743.540042:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5782, 5782] 00020000:00200000:0.0:1789760743.540052:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23683072 chunk: [23683072, 23687168) 23687168, 4096 00020000:00200000:0.0:1789760743.540058:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760743.540080:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5782, 5782] 00000008:00010000:0.0:1789760743.540099:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760743.540105:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760743.540117:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23683072, 23687168) append 0 append lockpos 0 00000080:00200000:0.0:1789760743.540130:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23683072=0x1696000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760743.540967:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760743.540974:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23687168 00000020:00010000:0.0:1789760743.540985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760743.540989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.540993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.540997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760743.541000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760743.541011:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760743.541019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760743.541046:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760743.541213:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 23683072<>23687168 00000100:00100000:1.0:1789760743.541434:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.541561:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820800:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760743.590668:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973099 gen 2 00000008:00010000:1.0:1789760743.590734:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760743.590743:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.590754:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.590775:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760743.590805:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820800:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760743.590906:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5782, 5782] 00000080:00200000:0.0:1789760743.590931:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760743.590938:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760743.590954:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23687168 i_blocks 47111 00000020:00200000:0.0:1789760743.590957:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760743.590966:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760743.591000:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23687168, count: 4096 00000080:00200000:0.0:1789760743.591007:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23687168, bytes: 4096 00000080:00200000:0.0:1789760743.591017:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760743.591026:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760743.591029:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23687168, 23691264) 0 0 00000080:00200000:0.0:1789760743.591034:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760743.591051:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5783, 5783] 00020000:00200000:0.0:1789760743.591061:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23687168 chunk: [23687168, 23691264) 23691264, 4096 00020000:00200000:0.0:1789760743.591066:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760743.591086:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5783, 5783] 00000008:00010000:0.0:1789760743.591104:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760743.591110:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760743.591121:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23687168, 23691264) append 0 append lockpos 0 00000080:00200000:0.0:1789760743.591136:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23687168=0x1697000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760743.591244:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760743.591247:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23691264 00000020:00010000:0.0:1789760743.591256:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760743.591260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.591263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760743.591266:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760743.591269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760743.591275:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760743.591282:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760743.591298:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760743.591770:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 23687168<>23691264 00000100:00100000:1.0:1789760743.591830:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820928:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760743.652049:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973100 gen 2 00000008:00010000:1.0:1789760743.652101:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760743.652109:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.652120:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.652462:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760743.652495:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495820928:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.652616:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5783, 5783] 00000080:00200000:2.0:1789760743.652633:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.652641:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.652657:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23691264 i_blocks 47111 00000020:00200000:2.0:1789760743.652661:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.652670:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.652705:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23691264, count: 4096 00000080:00200000:2.0:1789760743.652711:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23691264, bytes: 4096 00000080:00200000:2.0:1789760743.652722:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.652731:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.652733:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23691264, 23695360) 0 0 00000080:00200000:2.0:1789760743.652737:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.652754:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5784, 5784] 00020000:00200000:2.0:1789760743.652763:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23691264 chunk: [23691264, 23695360) 23695360, 4096 00020000:00200000:2.0:1789760743.652768:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.652787:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5784, 5784] 00000008:00010000:2.0:1789760743.652804:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.652810:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.652822:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23691264, 23695360) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.652836:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23691264=0x1698000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.652891:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.652895:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23695360 00000020:00010000:2.0:1789760743.652903:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.652907:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.652910:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.652913:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.652917:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.652924:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.652931:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.652947:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760743.653070:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 23691264<>23695360 00000100:00100000:3.0:1789760743.653161:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495821056:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760743.701153:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973101 gen 2 00000008:00010000:3.0:1789760743.701233:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760743.701353:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.701365:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.701496:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760743.701524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495821056:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.701794:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5784, 5784] 00000080:00200000:2.0:1789760743.701813:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.701819:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.701837:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23695360 i_blocks 47111 00000020:00200000:2.0:1789760743.701841:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.701851:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.701887:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23695360, count: 4096 00000080:00200000:2.0:1789760743.701895:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23695360, bytes: 4096 00000080:00200000:2.0:1789760743.701908:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.703205:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.703209:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23695360, 23699456) 0 0 00000080:00200000:2.0:1789760743.703216:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.703240:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5785, 5785] 00020000:00200000:2.0:1789760743.703938:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23695360 chunk: [23695360, 23699456) 23699456, 4096 00020000:00200000:2.0:1789760743.703944:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.703974:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5785, 5785] 00000008:00010000:2.0:1789760743.704006:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.704012:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.704027:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23695360, 23699456) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.704044:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23695360=0x1699000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.704129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.704133:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23699456 00000020:00010000:2.0:1789760743.704142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.704146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.704149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.704153:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.704155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.704213:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.704221:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.704264:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760743.704427:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 23695360<>23699456 00000100:00100000:3.0:1789760743.704829:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821184:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760743.765116:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973102 gen 2 00000008:00010000:3.0:1789760743.765208:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760743.765217:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.765232:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.765256:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760743.765288:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821184:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.765291:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5785, 5785] 00000080:00200000:2.0:1789760743.765311:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.765319:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.765335:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23699456 i_blocks 47111 00000020:00200000:2.0:1789760743.765339:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.765351:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.765384:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23699456, count: 4096 00000080:00200000:2.0:1789760743.765391:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23699456, bytes: 4096 00000080:00200000:2.0:1789760743.765401:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.765409:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.765411:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23699456, 23703552) 0 0 00000080:00200000:2.0:1789760743.765415:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.765431:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5786, 5786] 00020000:00200000:2.0:1789760743.765439:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23699456 chunk: [23699456, 23703552) 23703552, 4096 00020000:00200000:2.0:1789760743.765443:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.765461:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5786, 5786] 00000008:00010000:2.0:1789760743.765480:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.765486:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.765497:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23699456, 23703552) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.765511:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23699456=0x169a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.765576:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.765579:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23703552 00000020:00010000:2.0:1789760743.765715:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.765719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.765723:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.765727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.765730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.765738:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.765746:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.765773:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760743.765919:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 23699456<>23703552 00000100:00100000:3.0:1789760743.766020:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760743.766037:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821312:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760743.798702:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760743.798719:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.798731:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.798756:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760743.798785:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821312:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.798943:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5786, 5786] 00000080:00200000:2.0:1789760743.798960:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.798968:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.798984:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23703552 i_blocks 47111 00000020:00200000:2.0:1789760743.798997:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.799007:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.799049:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23703552, count: 4096 00000080:00200000:2.0:1789760743.799058:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23703552, bytes: 4096 00000080:00200000:2.0:1789760743.799069:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.799080:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.799083:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23703552, 23707648) 0 0 00000080:00200000:2.0:1789760743.799087:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.799106:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5787, 5787] 00020000:00200000:2.0:1789760743.799114:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23703552 chunk: [23703552, 23707648) 23707648, 4096 00020000:00200000:2.0:1789760743.799118:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.799138:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5787, 5787] 00000008:00010000:2.0:1789760743.799156:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.799162:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.799208:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23703552, 23707648) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.799223:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23703552=0x169b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.799277:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.799280:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23707648 00000020:00010000:2.0:1789760743.799289:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.799293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.799295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.799299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.799302:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.799310:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.799317:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.799334:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760743.799455:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 23703552<>23707648 00000100:00100000:3.0:1789760743.799514:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821440:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760743.827509:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973104 gen 2 00000008:00010000:3.0:1789760743.827672:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760743.827681:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.827692:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.827718:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760743.827749:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821440:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760743.827876:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5787, 5787] 00000080:00200000:2.0:1789760743.827893:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.827900:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.828158:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23707648 i_blocks 47111 00000020:00200000:2.0:1789760743.828164:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.828208:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.828246:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23707648, count: 4096 00000080:00200000:2.0:1789760743.828253:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23707648, bytes: 4096 00000080:00200000:2.0:1789760743.828265:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.828274:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.828276:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23707648, 23711744) 0 0 00000080:00200000:2.0:1789760743.828280:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.828298:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5788, 5788] 00020000:00200000:2.0:1789760743.828307:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23707648 chunk: [23707648, 23711744) 23711744, 4096 00020000:00200000:2.0:1789760743.828312:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.828333:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5788, 5788] 00000008:00010000:2.0:1789760743.828353:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.828359:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760743.828372:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23707648, 23711744) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.828387:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23707648=0x169c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.828444:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.828447:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23711744 00000020:00010000:2.0:1789760743.828456:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.828460:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.828463:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.828466:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.828470:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.828477:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.828484:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.828502:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760743.829484:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 23707648<>23711744 00000100:00100000:3.0:1789760743.829656:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760743.829683:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821568:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760743.861233:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973105 gen 2 00000008:00010000:3.0:1789760743.861286:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760743.861294:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.861304:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.861330:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760743.861360:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821568:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760743.861371:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5788, 5788] 00000080:00200000:1.0:1789760743.861391:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.861398:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.861414:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23711744 i_blocks 47111 00000020:00200000:1.0:1789760743.861418:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.861430:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.861461:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23711744, count: 4096 00000080:00200000:1.0:1789760743.861469:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23711744, bytes: 4096 00000080:00200000:1.0:1789760743.861479:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.861488:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.861491:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23711744, 23715840) 0 0 00000080:00200000:1.0:1789760743.861495:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.861513:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5789, 5789] 00020000:00200000:1.0:1789760743.861526:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23711744 chunk: [23711744, 23715840) 23715840, 4096 00020000:00200000:1.0:1789760743.861531:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.861553:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5789, 5789] 00000008:00010000:1.0:1789760743.861571:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.861578:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760743.861601:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23711744, 23715840) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.861615:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23711744=0x169d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.861673:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.861676:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23715840 00000020:00010000:1.0:1789760743.861685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.861688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.861691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.861695:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088144e8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.861698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.861705:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.861711:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.861726:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760743.861854:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 23711744<>23715840 00000100:00100000:1.0:1789760743.861903:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.861919:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495821696:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760743.884237:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760743.896905:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760743.896979:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.897050:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.897116:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760743.897150:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495821696:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760743.897322:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5789, 5789] 00000080:00200000:3.0:1789760743.897372:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760743.897379:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760743.897394:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23715840 i_blocks 47111 00000020:00200000:3.0:1789760743.897398:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760743.897406:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760743.897459:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23715840, count: 4096 00000080:00200000:3.0:1789760743.897468:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23715840, bytes: 4096 00000080:00200000:3.0:1789760743.897479:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760743.897489:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760743.897491:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23715840, 23719936) 0 0 00000080:00200000:3.0:1789760743.897496:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760743.897512:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5790, 5790] 00020000:00200000:3.0:1789760743.897521:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23715840 chunk: [23715840, 23719936) 23719936, 4096 00020000:00200000:3.0:1789760743.897526:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760743.897546:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5790, 5790] 00000008:00010000:3.0:1789760743.897566:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760743.897647:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760743.897660:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23715840, 23719936) append 0 append lockpos 0 00000080:00200000:3.0:1789760743.897675:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23715840=0x169e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760743.897737:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760743.897740:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23719936 00000020:00010000:3.0:1789760743.897750:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760743.897753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760743.897756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760743.897760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760743.897762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760743.897769:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760743.897776:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760743.897793:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760743.898116:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 23715840<>23719936 00000100:00100000:2.0:1789760743.898210:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821824:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760743.941620:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973107 gen 2 00000008:00010000:2.0:1789760743.941686:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:2.0:1789760743.941694:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760743.941705:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760743.941775:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:2.0:1789760743.941805:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821824:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760743.942419:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5790, 5790] 00000080:00200000:3.0:1789760743.942437:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760743.942444:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760743.942461:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23719936 i_blocks 47111 00000020:00200000:3.0:1789760743.942466:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760743.942476:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760743.942511:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23719936, count: 4096 00000080:00200000:3.0:1789760743.942519:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23719936, bytes: 4096 00000080:00200000:3.0:1789760743.942529:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760743.942539:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760743.942542:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23719936, 23724032) 0 0 00000080:00200000:3.0:1789760743.942547:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760743.942564:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5791, 5791] 00020000:00200000:3.0:1789760743.942713:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23719936 chunk: [23719936, 23724032) 23724032, 4096 00020000:00200000:3.0:1789760743.942719:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760743.942742:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5791, 5791] 00000008:00010000:3.0:1789760743.942761:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760743.942766:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760743.942778:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23719936, 23724032) append 0 append lockpos 0 00000080:00200000:3.0:1789760743.942792:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23719936=0x169f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760743.943021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760743.943024:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23724032 00000020:00010000:3.0:1789760743.943034:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760743.943039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760743.943043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760743.943048:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760743.943050:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760743.943058:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760743.943065:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760743.943086:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760743.943304:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 23719936<>23724032 00000100:00100000:3.0:1789760743.943360:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760743.943380:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821952:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760744.008056:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:3.0:1789760744.008076:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.008088:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.008898:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:3.0:1789760744.008946:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495821952:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760744.009555:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5791, 5791] 00000080:00200000:1.0:1789760744.009578:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.009585:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.009750:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23724032 i_blocks 47111 00000020:00200000:1.0:1789760744.010024:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.010040:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.010080:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23724032, count: 4096 00000080:00200000:1.0:1789760744.010089:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23724032, bytes: 4096 00000080:00200000:1.0:1789760744.010104:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.010114:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.010117:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23724032, 23728128) 0 0 00000080:00200000:1.0:1789760744.010122:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.010303:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5792, 5792] 00020000:00200000:1.0:1789760744.010383:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23724032 chunk: [23724032, 23728128) 23728128, 4096 00020000:00200000:1.0:1789760744.010389:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.010415:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5792, 5792] 00000008:00010000:1.0:1789760744.010439:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.010447:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760744.010536:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23724032, 23728128) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.010561:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23724032=0x16a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.011045:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.011049:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23728128 00000020:00010000:1.0:1789760744.011059:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.011062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.011066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.011070:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.011072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.011082:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.011089:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.011116:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760744.012013:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 23724032<>23728128 00000100:00100000:0.0:1789760744.012095:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760744.012563:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495822080:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.061637:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973109 gen 2 00000008:00010000:1.0:1789760744.061703:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760744.061712:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.061724:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.061745:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760744.061886:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495822080:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760744.062069:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5792, 5792] 00000080:00200000:0.0:1789760744.062085:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760744.062091:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760744.062107:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23728128 i_blocks 47111 00000020:00200000:0.0:1789760744.062110:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760744.062119:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760744.062151:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23728128, count: 4096 00000080:00200000:0.0:1789760744.062157:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23728128, bytes: 4096 00000080:00200000:0.0:1789760744.062201:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760744.062211:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760744.062214:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23728128, 23732224) 0 0 00000080:00200000:0.0:1789760744.062218:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760744.062234:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5793, 5793] 00020000:00200000:0.0:1789760744.062243:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23728128 chunk: [23728128, 23732224) 23732224, 4096 00020000:00200000:0.0:1789760744.062248:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760744.062267:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5793, 5793] 00000008:00010000:0.0:1789760744.062284:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760744.062290:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760744.062301:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23728128, 23732224) append 0 append lockpos 0 00000080:00200000:0.0:1789760744.062314:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23728128=0x16a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760744.062368:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760744.062371:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23732224 00000020:00010000:0.0:1789760744.062379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760744.062383:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.062386:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.062389:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760744.062392:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760744.062400:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760744.062406:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760744.062423:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760744.063724:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 23728128<>23732224 00000100:00100000:1.0:1789760744.064548:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760744.064586:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495822208:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760744.119002:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760744.119018:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.119031:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760744.119290:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5793, 5793] 00000008:00010000:1.0:1789760744.119442:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000080:00200000:0.0:1789760744.119540:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760744.119548:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760744.119567:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23732224 i_blocks 47111 00000020:00200000:0.0:1789760744.119571:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760744.119581:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:1.0:1789760744.119758:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495822208:192.168.203.109@tcp:4:dd.0 00000080:00200008:0.0:1789760744.119883:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23732224, count: 4096 00000080:00200000:0.0:1789760744.119892:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23732224, bytes: 4096 00000080:00200000:0.0:1789760744.119908:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760744.119917:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760744.119920:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23732224, 23736320) 0 0 00000080:00200000:0.0:1789760744.119924:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760744.119942:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5794, 5794] 00020000:00200000:0.0:1789760744.119951:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23732224 chunk: [23732224, 23736320) 23736320, 4096 00020000:00200000:0.0:1789760744.119956:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760744.119977:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5794, 5794] 00000008:00010000:0.0:1789760744.119997:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760744.120002:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760744.120015:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23732224, 23736320) append 0 append lockpos 0 00000080:00200000:0.0:1789760744.120405:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23732224=0x16a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760744.120549:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760744.120553:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23736320 00000020:00010000:0.0:1789760744.120562:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760744.120566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.120569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.120573:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760744.120576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760744.120596:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760744.120604:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760744.120626:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760744.120761:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 23732224<>23736320 00000100:00100000:1.0:1789760744.120888:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760744.120914:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495822336:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.165198:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973111 gen 2 00000008:00010000:1.0:1789760744.165264:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760744.165275:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.165285:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.165308:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760744.165338:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495822336:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760744.165531:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5794, 5794] 00000080:00200000:0.0:1789760744.165547:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760744.165554:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760744.165570:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23736320 i_blocks 47111 00000020:00200000:0.0:1789760744.165574:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760744.165583:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760744.165660:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23736320, count: 4096 00000080:00200000:0.0:1789760744.165668:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23736320, bytes: 4096 00000080:00200000:0.0:1789760744.165678:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760744.165686:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760744.165689:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23736320, 23740416) 0 0 00000080:00200000:0.0:1789760744.165694:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760744.165711:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5795, 5795] 00020000:00200000:0.0:1789760744.165718:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23736320 chunk: [23736320, 23740416) 23740416, 4096 00020000:00200000:0.0:1789760744.165723:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760744.165743:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5795, 5795] 00000008:00010000:0.0:1789760744.165763:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760744.165769:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760744.165780:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23736320, 23740416) append 0 append lockpos 0 00000080:00200000:0.0:1789760744.165794:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23736320=0x16a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760744.165851:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760744.165854:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23740416 00000020:00010000:0.0:1789760744.165863:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760744.165867:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.165870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.165873:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760744.165877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760744.165885:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760744.165892:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760744.165908:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760744.166034:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 23736320<>23740416 00000100:00100000:1.0:1789760744.168476:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760744.168513:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495822464:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.206473:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973112 gen 2 00000008:00010000:1.0:1789760744.206519:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760744.206529:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.206540:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.206563:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760744.206603:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495822464:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760744.206711:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5795, 5795] 00000080:00200000:0.0:1789760744.206732:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760744.206738:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760744.206755:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23740416 i_blocks 47111 00000020:00200000:0.0:1789760744.206757:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760744.206766:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760744.206797:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23740416, count: 4096 00000080:00200000:0.0:1789760744.206803:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23740416, bytes: 4096 00000080:00200000:0.0:1789760744.206811:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760744.206818:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760744.206820:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23740416, 23744512) 0 0 00000080:00200000:0.0:1789760744.206824:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760744.206839:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5796, 5796] 00020000:00200000:0.0:1789760744.206846:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23740416 chunk: [23740416, 23744512) 23744512, 4096 00020000:00200000:0.0:1789760744.206849:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760744.206952:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5796, 5796] 00000008:00010000:0.0:1789760744.206972:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760744.207755:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760744.207777:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23740416, 23744512) append 0 append lockpos 0 00000080:00200000:0.0:1789760744.207793:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23740416=0x16a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760744.208022:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760744.208025:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23744512 00000020:00010000:0.0:1789760744.208088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760744.208093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.208097:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.208101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760744.208104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760744.208114:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760744.208122:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760744.208149:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760744.208406:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 23740416<>23744512 00000100:00100000:1.0:1789760744.208510:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760744.208530:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495822592:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.247395:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973113 gen 2 00000008:00010000:1.0:1789760744.247447:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760744.247455:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.247467:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.247498:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760744.247527:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495822592:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760744.247564:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5796, 5796] 00000080:00200000:0.0:1789760744.247584:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760744.247601:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760744.247617:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23744512 i_blocks 47111 00000020:00200000:0.0:1789760744.247620:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760744.247630:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760744.247662:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23744512, count: 4096 00000080:00200000:0.0:1789760744.247670:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23744512, bytes: 4096 00000080:00200000:0.0:1789760744.247683:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760744.247692:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760744.247695:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23744512, 23748608) 0 0 00000080:00200000:0.0:1789760744.247700:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760744.247716:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5797, 5797] 00020000:00200000:0.0:1789760744.247725:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23744512 chunk: [23744512, 23748608) 23748608, 4096 00020000:00200000:0.0:1789760744.247729:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760744.247749:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5797, 5797] 00000008:00010000:0.0:1789760744.247766:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760744.247771:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760744.247783:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23744512, 23748608) append 0 append lockpos 0 00000080:00200000:0.0:1789760744.247798:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23744512=0x16a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760744.247850:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760744.247853:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23748608 00000020:00010000:0.0:1789760744.247861:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760744.247865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.247868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.247871:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760744.247875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760744.247882:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760744.247889:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.248822:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:1.0:1789760744.248990:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 23744512<>23748608 00000100:00100000:1.0:1789760744.249071:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495822720:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.299625:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973114 gen 2 00000008:00010000:1.0:1789760744.299677:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760744.299684:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.299695:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.299714:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760744.299741:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495822720:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760744.300354:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5797, 5797] 00000080:00200000:0.0:1789760744.300371:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760744.300387:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760744.300403:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23748608 i_blocks 47111 00000020:00200000:0.0:1789760744.300407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760744.300427:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760744.300462:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23748608, count: 4096 00000080:00200000:0.0:1789760744.300469:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23748608, bytes: 4096 00000080:00200000:0.0:1789760744.300480:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760744.300490:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760744.300492:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23748608, 23752704) 0 0 00000080:00200000:0.0:1789760744.300497:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760744.300515:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5798, 5798] 00020000:00200000:0.0:1789760744.300524:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23748608 chunk: [23748608, 23752704) 23752704, 4096 00020000:00200000:0.0:1789760744.300529:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760744.300551:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5798, 5798] 00000008:00010000:0.0:1789760744.300579:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760744.300594:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760744.300607:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23748608, 23752704) append 0 append lockpos 0 00000080:00200000:0.0:1789760744.300622:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23748608=0x16a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760744.300681:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760744.300684:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23752704 00000020:00010000:0.0:1789760744.300694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760744.300697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.300701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.300704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760744.300707:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760744.300715:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760744.300721:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760744.300736:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760744.300856:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 23748608<>23752704 00000100:00100000:1.0:1789760744.301341:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495822848:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.338516:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973115 gen 2 00000008:00010000:1.0:1789760744.338573:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760744.338582:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.338604:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.338625:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760744.338654:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495822848:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760744.338798:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5798, 5798] 00000080:00200000:0.0:1789760744.338816:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760744.338823:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760744.338839:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23752704 i_blocks 47111 00000020:00200000:0.0:1789760744.338843:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760744.338864:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760744.338898:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23752704, count: 4096 00000080:00200000:0.0:1789760744.338905:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23752704, bytes: 4096 00000080:00200000:0.0:1789760744.338915:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760744.338924:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760744.338926:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23752704, 23756800) 0 0 00000080:00200000:0.0:1789760744.338931:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760744.338948:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5799, 5799] 00020000:00200000:0.0:1789760744.338956:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23752704 chunk: [23752704, 23756800) 23756800, 4096 00020000:00200000:0.0:1789760744.338960:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760744.338981:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5799, 5799] 00000008:00010000:0.0:1789760744.338998:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760744.339004:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760744.339022:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23752704, 23756800) append 0 append lockpos 0 00000080:00200000:0.0:1789760744.339035:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23752704=0x16a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760744.339098:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760744.339101:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23756800 00000020:00010000:0.0:1789760744.339109:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760744.339113:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.339116:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760744.339119:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760744.339122:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760744.339129:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760744.339136:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760744.339305:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760744.339428:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 23752704<>23756800 00000100:00100000:0.0:1789760744.339471:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760744.339486:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495822976:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.401065:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973116 gen 2 00000008:00010000:1.0:1789760744.401114:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760744.401122:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.401133:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.401162:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760744.401227:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495822976:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760744.401393:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5799, 5799] 00000080:00200000:3.0:1789760744.401817:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.401824:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.401840:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23756800 i_blocks 47111 00000020:00200000:3.0:1789760744.401844:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.401863:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.401900:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23756800, count: 4096 00000080:00200000:3.0:1789760744.401915:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23756800, bytes: 4096 00000080:00200000:3.0:1789760744.401927:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.401936:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.401939:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23756800, 23760896) 0 0 00000080:00200000:3.0:1789760744.401944:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.401961:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5800, 5800] 00020000:00200000:3.0:1789760744.401969:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23756800 chunk: [23756800, 23760896) 23760896, 4096 00020000:00200000:3.0:1789760744.401973:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.401993:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5800, 5800] 00000008:00010000:3.0:1789760744.402011:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.402018:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760744.402030:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23756800, 23760896) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.402044:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23756800=0x16a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.402107:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.402110:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23760896 00000020:00010000:3.0:1789760744.402118:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.402122:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.402125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.402128:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.402131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.402138:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.402382:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.402402:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760744.402527:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 23756800<>23760896 00000100:00100000:2.0:1789760744.403796:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495823104:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760744.443882:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973117 gen 2 00000008:00010000:2.0:1789760744.443935:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760744.443944:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760744.443957:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760744.443984:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:2.0:1789760744.444016:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495823104:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760744.445286:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5800, 5800] 00000080:00200000:3.0:1789760744.445311:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.445317:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.445334:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23760896 i_blocks 47111 00000020:00200000:3.0:1789760744.445339:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.445352:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.445389:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23760896, count: 4096 00000080:00200000:3.0:1789760744.445396:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23760896, bytes: 4096 00000080:00200000:3.0:1789760744.445412:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.445421:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.445424:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23760896, 23764992) 0 0 00000080:00200000:3.0:1789760744.445428:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.445446:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5801, 5801] 00020000:00200000:3.0:1789760744.445455:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23760896 chunk: [23760896, 23764992) 23764992, 4096 00020000:00200000:3.0:1789760744.445460:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.445482:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5801, 5801] 00000008:00010000:3.0:1789760744.445502:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.445508:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760744.445521:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23760896, 23764992) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.445537:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23760896=0x16a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.446819:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.446828:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23764992 00000020:00010000:3.0:1789760744.446841:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.446845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.446849:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.446853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.446855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.446866:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.446876:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.446905:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760744.447043:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 23760896<>23764992 00000100:00100000:2.0:1789760744.448263:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495823232:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760744.491873:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760744.491890:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760744.491901:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760744.491924:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760744.491957:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495823232:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760744.492048:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5801, 5801] 00000080:00200000:3.0:1789760744.492064:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.492259:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.492275:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23764992 i_blocks 47111 00000020:00200000:3.0:1789760744.492279:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.492288:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.492848:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23764992, count: 4096 00000080:00200000:3.0:1789760744.492860:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23764992, bytes: 4096 00000080:00200000:3.0:1789760744.492872:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.492890:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.492893:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23764992, 23769088) 0 0 00000080:00200000:3.0:1789760744.492899:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.492917:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5802, 5802] 00020000:00200000:3.0:1789760744.492926:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23764992 chunk: [23764992, 23769088) 23769088, 4096 00020000:00200000:3.0:1789760744.493028:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.493051:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5802, 5802] 00000008:00010000:3.0:1789760744.493070:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.493076:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760744.493087:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23764992, 23769088) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.493101:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23764992=0x16aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.493665:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.493669:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23769088 00000020:00010000:3.0:1789760744.493680:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.493684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.493687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.493690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.493693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.493703:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.493721:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.493741:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760744.494651:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 23764992<>23769088 00000100:00100000:3.0:1789760744.494724:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495823360:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760744.545681:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973118 gen 2 00000100:00100000:3.0:1789760744.545720:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203145180 x1876699495823360/t4294973119(4294973119) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760760 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760744.545767:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:3.0:1789760744.545774:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.545784:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.545807:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:3.0:1789760744.545830:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495823360:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760744.546977:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5802, 5802] 00000080:00200000:2.0:1789760744.546996:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.547003:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.547017:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23769088 i_blocks 47111 00000020:00200000:2.0:1789760744.547021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.547032:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.547075:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23769088, count: 4096 00000080:00200000:2.0:1789760744.547084:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23769088, bytes: 4096 00000080:00200000:2.0:1789760744.547097:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.547106:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.547108:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23769088, 23773184) 0 0 00000080:00200000:2.0:1789760744.547112:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.547127:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5803, 5803] 00020000:00200000:2.0:1789760744.547134:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23769088 chunk: [23769088, 23773184) 23773184, 4096 00020000:00200000:2.0:1789760744.547139:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.547159:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5803, 5803] 00000008:00010000:2.0:1789760744.547368:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.547375:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760744.547387:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23769088, 23773184) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.547401:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23769088=0x16ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.547458:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.547462:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23773184 00000020:00010000:2.0:1789760744.547470:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.547474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.547477:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.547480:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.547483:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.547492:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.547499:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.547517:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760744.547722:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 23769088<>23773184 00000100:00100000:3.0:1789760744.547808:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495823488:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760744.580367:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973119 gen 2 00000100:00100000:2.0:1789760744.580413:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264e00 x1876699495823488/t4294973120(4294973120) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760760 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760744.580462:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760744.580469:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760744.580479:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760744.580498:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760744.580523:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495823488:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760744.580697:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5803, 5803] 00000080:00200000:1.0:1789760744.580715:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.580721:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.580737:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23773184 i_blocks 47111 00000020:00200000:1.0:1789760744.580741:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.580750:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.580785:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23773184, count: 4096 00000080:00200000:1.0:1789760744.580793:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23773184, bytes: 4096 00000080:00200000:1.0:1789760744.580804:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.580813:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.580816:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23773184, 23777280) 0 0 00000080:00200000:1.0:1789760744.580820:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.580836:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5804, 5804] 00020000:00200000:1.0:1789760744.580845:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23773184 chunk: [23773184, 23777280) 23777280, 4096 00020000:00200000:1.0:1789760744.580850:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.580867:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5804, 5804] 00000008:00010000:1.0:1789760744.580884:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.580891:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760744.580901:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23773184, 23777280) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.580915:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23773184=0x16ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.580968:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.580971:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23777280 00000020:00010000:1.0:1789760744.580980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.580983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.580986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.580989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.580992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.581000:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.581007:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.581033:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760744.581152:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 23773184<>23777280 00000100:00100000:1.0:1789760744.581229:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760744.581246:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495823616:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.630889:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973121 gen 2 00000008:00010000:1.0:1789760744.630966:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760744.630976:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.630987:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.631073:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760744.631105:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495823616:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760744.631656:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5804, 5804] 00000080:00200000:2.0:1789760744.631675:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.631681:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.631833:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23777280 i_blocks 47111 00000020:00200000:2.0:1789760744.631837:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.631849:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.631888:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23777280, count: 4096 00000080:00200000:2.0:1789760744.631895:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23777280, bytes: 4096 00000080:00200000:2.0:1789760744.631910:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.631919:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.631922:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23777280, 23781376) 0 0 00000080:00200000:2.0:1789760744.631926:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.631943:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5805, 5805] 00020000:00200000:2.0:1789760744.631952:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23777280 chunk: [23777280, 23781376) 23781376, 4096 00020000:00200000:2.0:1789760744.631957:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.631980:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5805, 5805] 00000008:00010000:2.0:1789760744.632000:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.632006:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760744.632019:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23777280, 23781376) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.632034:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23777280=0x16ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.632103:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.632107:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23781376 00000020:00010000:2.0:1789760744.632116:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.632120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.632123:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.632127:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.632130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.632139:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.632148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.632383:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760744.632848:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 23777280<>23781376 00000100:00100000:2.0:1789760744.632910:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495823744:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760744.681506:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973122 gen 2 00000008:00010000:2.0:1789760744.681559:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760744.681568:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760744.681580:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760744.681725:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760744.681756:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495823744:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760744.681839:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5805, 5805] 00000080:00200000:3.0:1789760744.681865:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.681872:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.681889:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23781376 i_blocks 47111 00000020:00200000:3.0:1789760744.681893:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.681903:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.681936:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23781376, count: 4096 00000080:00200000:3.0:1789760744.681944:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23781376, bytes: 4096 00000080:00200000:3.0:1789760744.681956:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.681965:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.681967:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23781376, 23785472) 0 0 00000080:00200000:3.0:1789760744.681972:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.681989:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5806, 5806] 00020000:00200000:3.0:1789760744.681997:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23781376 chunk: [23781376, 23785472) 23785472, 4096 00020000:00200000:3.0:1789760744.682001:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.682022:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5806, 5806] 00000008:00010000:3.0:1789760744.682039:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.682044:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760744.682055:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23781376, 23785472) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.682070:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23781376=0x16ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.682123:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.682126:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23785472 00000020:00010000:3.0:1789760744.682135:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.682139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.682142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.682145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.682149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.682156:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.682162:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.682497:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760744.682636:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 23781376<>23785472 00000100:00100000:3.0:1789760744.682747:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760744.682765:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495823872:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760744.732453:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973123 gen 2 00000008:00010000:3.0:1789760744.732618:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:3.0:1789760744.732626:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.732644:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.732671:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:3.0:1789760744.732703:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495823872:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760744.733973:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5806, 5806] 00000080:00200000:1.0:1789760744.733998:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.734005:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.734022:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23785472 i_blocks 47111 00000020:00200000:1.0:1789760744.734027:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.734056:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.734099:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23785472, count: 4096 00000080:00200000:1.0:1789760744.734108:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23785472, bytes: 4096 00000080:00200000:1.0:1789760744.734123:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.734134:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.734136:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23785472, 23789568) 0 0 00000080:00200000:1.0:1789760744.734141:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.734158:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5807, 5807] 00020000:00200000:1.0:1789760744.734166:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23785472 chunk: [23785472, 23789568) 23789568, 4096 00020000:00200000:1.0:1789760744.734205:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.734232:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5807, 5807] 00000008:00010000:1.0:1789760744.734253:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.734259:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760744.734273:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23785472, 23789568) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.734290:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23785472=0x16af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.734360:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.734363:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23789568 00000020:00010000:1.0:1789760744.734372:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.734375:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.734378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.734394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.734397:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.734405:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.734412:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.734432:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760744.734570:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 23785472<>23789568 00000100:00100000:0.0:1789760744.735990:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760744.736025:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495824000:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760744.789695:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973124 gen 2 00000008:00010000:1.0:1789760744.789747:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760744.789755:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.789767:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.789788:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760744.789817:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495824000:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760744.789926:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5807, 5807] 00000080:00200000:3.0:1789760744.789943:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.789949:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.789966:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23789568 i_blocks 47111 00000020:00200000:3.0:1789760744.789970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.789979:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.790015:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23789568, count: 4096 00000080:00200000:3.0:1789760744.790022:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23789568, bytes: 4096 00000080:00200000:3.0:1789760744.790033:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.790042:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.790056:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23789568, 23793664) 0 0 00000080:00200000:3.0:1789760744.790061:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.790078:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5808, 5808] 00020000:00200000:3.0:1789760744.790085:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23789568 chunk: [23789568, 23793664) 23793664, 4096 00020000:00200000:3.0:1789760744.790091:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.790111:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5808, 5808] 00000008:00010000:3.0:1789760744.790129:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.790134:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760744.790146:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23789568, 23793664) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.790161:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23789568=0x16b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.790246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.790249:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23793664 00000020:00010000:3.0:1789760744.790259:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.790263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.790267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.790270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.790273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.790280:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.790288:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.790305:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760744.790538:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 23789568<>23793664 00000100:00100000:2.0:1789760744.790620:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495824128:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760744.828820:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760744.828845:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760744.828856:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760744.828878:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760744.828912:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495824128:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760744.828923:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5808, 5808] 00000080:00200000:3.0:1789760744.828943:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.828950:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.828969:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23793664 i_blocks 47111 00000020:00200000:3.0:1789760744.828973:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.829499:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.830127:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23793664, count: 4096 00000080:00200000:2.0:1789760744.830139:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23793664, bytes: 4096 00000080:00200000:2.0:1789760744.830153:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.830162:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.830165:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23793664, 23797760) 0 0 00000080:00200000:2.0:1789760744.830208:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.830229:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5809, 5809] 00020000:00200000:2.0:1789760744.830239:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23793664 chunk: [23793664, 23797760) 23797760, 4096 00020000:00200000:2.0:1789760744.830245:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.830268:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5809, 5809] 00000008:00010000:2.0:1789760744.830287:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.830293:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760744.830307:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23793664, 23797760) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.830320:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23793664=0x16b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.831158:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.831162:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23797760 00000020:00010000:2.0:1789760744.831667:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.831673:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.831677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.831682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814c68 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.831685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.831701:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.831712:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.831790:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760744.831975:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 23793664<>23797760 00000100:00100000:2.0:1789760744.833486:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760744.833532:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495824256:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760744.901641:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973126 gen 2 00000008:00010000:2.0:1789760744.901799:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760744.901808:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760744.901818:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760744.901878:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760744.901912:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495824256:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760744.904678:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5809, 5809] 00000080:00200000:3.0:1789760744.904706:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.904723:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.904743:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23797760 i_blocks 47111 00000020:00200000:3.0:1789760744.904749:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.904767:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.904842:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23797760, count: 4096 00000080:00200000:3.0:1789760744.904852:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23797760, bytes: 4096 00000080:00200000:3.0:1789760744.904868:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.904877:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.904879:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23797760, 23801856) 0 0 00000080:00200000:3.0:1789760744.906035:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.906060:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5810, 5810] 00020000:00200000:3.0:1789760744.906069:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23797760 chunk: [23797760, 23801856) 23801856, 4096 00020000:00200000:3.0:1789760744.906075:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.906100:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5810, 5810] 00000008:00010000:3.0:1789760744.906123:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.906130:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760744.906144:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23797760, 23801856) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.906160:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23797760=0x16b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.906679:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.906684:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23801856 00000020:00010000:3.0:1789760744.906693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.906696:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.906700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.906704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.906707:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.907655:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.907668:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.907696:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760744.908155:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 23797760<>23801856 00000100:00100000:2.0:1789760744.908652:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495824384:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760744.909201:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:2.0:1789760744.979112:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760744.979131:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760744.979144:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760744.979217:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760744.979252:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495824384:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760744.979365:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5810, 5810] 00000080:00200000:1.0:1789760744.979382:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.979388:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.979405:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23801856 i_blocks 47111 00000020:00200000:1.0:1789760744.979408:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.979419:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.979458:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23801856, count: 4096 00000080:00200000:1.0:1789760744.979466:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23801856, bytes: 4096 00000080:00200000:1.0:1789760744.979708:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.979719:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.979722:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23801856, 23805952) 0 0 00000080:00200000:1.0:1789760744.979728:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.979747:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5811, 5811] 00020000:00200000:1.0:1789760744.979757:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23801856 chunk: [23801856, 23805952) 23805952, 4096 00020000:00200000:1.0:1789760744.979763:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.979784:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5811, 5811] 00000008:00010000:1.0:1789760744.979804:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.979811:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760744.979824:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23801856, 23805952) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.979840:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23801856=0x16b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.979916:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.980124:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23805952 00000020:00010000:1.0:1789760744.980134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.980137:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.980141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.980144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.980148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.980155:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.980162:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.980228:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760744.980360:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 23801856<>23805952 00000100:00100000:0.0:1789760744.980471:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760744.980503:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495824512:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760745.048961:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973127 gen 2 00000100:00100000:1.0:1789760745.049818:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d500 x1876699495824512/t4294973128(4294973128) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760760 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760745.049896:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760745.049969:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.049980:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.050060:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760745.050081:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495824512:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.050158:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5811, 5811] 00000080:00200000:2.0:1789760745.050198:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.050205:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.050221:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23805952 i_blocks 47111 00000020:00200000:2.0:1789760745.050225:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.050236:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.050270:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23805952, count: 4096 00000080:00200000:2.0:1789760745.050278:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23805952, bytes: 4096 00000080:00200000:2.0:1789760745.050290:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.050300:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.050302:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23805952, 23810048) 0 0 00000080:00200000:2.0:1789760745.050307:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.050326:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5812, 5812] 00020000:00200000:2.0:1789760745.050338:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23805952 chunk: [23805952, 23810048) 23810048, 4096 00020000:00200000:2.0:1789760745.050343:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.050365:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5812, 5812] 00000008:00010000:2.0:1789760745.050385:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.050391:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.050403:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23805952, 23810048) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.050418:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23805952=0x16b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.050472:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.050475:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23810048 00000020:00010000:2.0:1789760745.050484:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.050487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.050490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.050494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.050497:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.050504:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.050512:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.050528:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760745.050682:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 23805952<>23810048 00000100:00100000:2.0:1789760745.050737:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760745.050757:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495824640:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760745.078809:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973129 gen 2 00000008:00010000:2.0:1789760745.078881:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760745.078889:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760745.078900:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760745.078993:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760745.079036:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495824640:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760745.082918:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5812, 5812] 00000080:00200000:3.0:1789760745.082951:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760745.082958:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760745.082980:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23810048 i_blocks 47111 00000020:00200000:3.0:1789760745.082985:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760745.083002:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760745.083038:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23810048, count: 4096 00000080:00200000:3.0:1789760745.083047:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23810048, bytes: 4096 00000080:00200000:3.0:1789760745.083062:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760745.083073:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760745.083075:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23810048, 23814144) 0 0 00000080:00200000:3.0:1789760745.083080:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760745.083098:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5813, 5813] 00020000:00200000:3.0:1789760745.083106:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23810048 chunk: [23810048, 23814144) 23814144, 4096 00020000:00200000:3.0:1789760745.083111:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760745.083143:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5813, 5813] 00000008:00010000:3.0:1789760745.083164:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760745.083196:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760745.083209:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23810048, 23814144) append 0 append lockpos 0 00000080:00200000:3.0:1789760745.083234:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23810048=0x16b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760745.083307:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760745.083322:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23814144 00000020:00010000:3.0:1789760745.083332:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:3.0:1789760745.083336:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760745.083339:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760745.083342:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760745.083345:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760745.083353:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760745.083359:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760745.083388:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760745.083532:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 23810048<>23814144 00000100:00100000:3.0:1789760745.083603:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760745.083628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495824768:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.126311:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973130 gen 2 00000008:00010000:2.0:1789760745.126476:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760745.126485:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760745.126496:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760745.126555:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760745.126596:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495824768:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760745.126684:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5813, 5813] 00000080:00200000:3.0:1789760745.126700:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760745.126705:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.126793:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23814144 i_blocks 47111 00000020:00200000:2.0:1789760745.126797:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.126808:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.126842:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23814144, count: 4096 00000080:00200000:2.0:1789760745.126848:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23814144, bytes: 4096 00000080:00200000:2.0:1789760745.126859:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.126868:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.126870:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23814144, 23818240) 0 0 00000080:00200000:2.0:1789760745.129283:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.129320:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5814, 5814] 00020000:00200000:2.0:1789760745.129333:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23814144 chunk: [23814144, 23818240) 23818240, 4096 00020000:00200000:2.0:1789760745.129339:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.129366:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5814, 5814] 00000008:00010000:2.0:1789760745.129391:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.129397:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.129409:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23814144, 23818240) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.129427:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23814144=0x16b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.129498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.129502:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23818240 00000020:00010000:2.0:1789760745.129511:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.129514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.129518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.129521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.129524:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.129535:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.129542:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.129617:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760745.129751:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 23814144<>23818240 00000100:00100000:3.0:1789760745.129855:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495824896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.214628:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973131 gen 2 00000008:00010000:3.0:1789760745.214679:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760745.214688:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.214701:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.214782:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760745.214810:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495824896:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.215060:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5814, 5814] 00000080:00200000:2.0:1789760745.215077:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.215083:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.215098:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23818240 i_blocks 47111 00000020:00200000:2.0:1789760745.215102:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.215111:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.215146:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23818240, count: 4096 00000080:00200000:2.0:1789760745.215153:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23818240, bytes: 4096 00000080:00200000:2.0:1789760745.215166:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.215201:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.215203:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23818240, 23822336) 0 0 00000080:00200000:2.0:1789760745.215208:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.215224:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5815, 5815] 00020000:00200000:2.0:1789760745.215232:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23818240 chunk: [23818240, 23822336) 23822336, 4096 00020000:00200000:2.0:1789760745.215237:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.215256:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5815, 5815] 00000008:00010000:2.0:1789760745.215273:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.215278:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.215290:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23818240, 23822336) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.215304:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23818240=0x16b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.215357:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.215360:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23822336 00000020:00010000:2.0:1789760745.215368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.215371:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.215374:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.215378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.215381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.215388:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.215396:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.215412:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760745.215531:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 23818240<>23822336 00000100:00100000:3.0:1789760745.215646:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760745.215663:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825024:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.264008:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973132 gen 2 00000008:00010000:3.0:1789760745.264062:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760745.264071:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.264082:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.264110:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000080:00200000:2.0:1789760745.264302:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5815, 5815] 00000080:00200000:2.0:1789760745.264321:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.264328:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760745.264332:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825024:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.264345:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23822336 i_blocks 47111 00000020:00200000:2.0:1789760745.264350:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.264361:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.264398:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23822336, count: 4096 00000080:00200000:2.0:1789760745.264405:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23822336, bytes: 4096 00000080:00200000:2.0:1789760745.264416:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.264425:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.264428:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23822336, 23826432) 0 0 00000080:00200000:2.0:1789760745.264432:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.264449:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5816, 5816] 00020000:00200000:2.0:1789760745.264458:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23822336 chunk: [23822336, 23826432) 23826432, 4096 00020000:00200000:2.0:1789760745.264463:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.264483:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5816, 5816] 00000008:00010000:2.0:1789760745.264502:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.264508:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.264520:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23822336, 23826432) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.264534:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23822336=0x16b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.264603:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.264607:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23826432 00000020:00010000:2.0:1789760745.264616:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.264619:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.264622:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.264626:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.264629:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.264636:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.264654:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.264670:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760745.266133:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 23822336<>23826432 00000100:00100000:3.0:1789760745.267944:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825152:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.303132:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973133 gen 2 00000008:00010000:3.0:1789760745.303230:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760745.303239:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.303252:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.303277:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760745.303311:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825152:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.303683:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5816, 5816] 00000080:00200000:2.0:1789760745.303701:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.303707:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.303722:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23826432 i_blocks 47111 00000020:00200000:2.0:1789760745.303725:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.303734:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.303764:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23826432, count: 4096 00000080:00200000:2.0:1789760745.303771:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23826432, bytes: 4096 00000080:00200000:2.0:1789760745.303780:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.303789:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.303791:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23826432, 23830528) 0 0 00000080:00200000:2.0:1789760745.303796:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.303813:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5817, 5817] 00020000:00200000:2.0:1789760745.303821:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23826432 chunk: [23826432, 23830528) 23830528, 4096 00020000:00200000:2.0:1789760745.303826:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.303843:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5817, 5817] 00000008:00010000:2.0:1789760745.303862:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.303868:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.303878:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23826432, 23830528) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.303893:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23826432=0x16b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.303945:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.303948:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23830528 00000020:00010000:2.0:1789760745.303956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.303959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.303962:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.303966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.303969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.303976:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.303982:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.303997:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760745.305620:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 23826432<>23830528 00000100:00100000:2.0:1789760745.305690:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760745.305713:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825280:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760745.343676:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973134 gen 2 00000008:00010000:2.0:1789760745.343729:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760745.343738:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760745.343809:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760745.343873:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760745.343913:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825280:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760745.344122:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5817, 5817] 00000080:00200000:3.0:1789760745.344139:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760745.344145:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.344271:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23830528 i_blocks 47111 00000020:00200000:2.0:1789760745.344276:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.344285:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.344319:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23830528, count: 4096 00000080:00200000:2.0:1789760745.344326:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23830528, bytes: 4096 00000080:00200000:2.0:1789760745.344338:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.344347:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.344349:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23830528, 23834624) 0 0 00000080:00200000:2.0:1789760745.344353:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.344372:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5818, 5818] 00020000:00200000:2.0:1789760745.344381:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23830528 chunk: [23830528, 23834624) 23834624, 4096 00020000:00200000:2.0:1789760745.344386:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.344406:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5818, 5818] 00000008:00010000:2.0:1789760745.344425:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.344431:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.344442:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23830528, 23834624) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.344456:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23830528=0x16ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.344509:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.344511:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23834624 00000020:00010000:2.0:1789760745.344518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.344522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.344525:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.344528:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.344531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.344538:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.344661:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.344709:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760745.344896:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 23830528<>23834624 00000100:00100000:3.0:1789760745.345031:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825408:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760745.377315:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973135 gen 2 00000008:00010000:2.0:1789760745.377370:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760745.377379:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760745.377393:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760745.377421:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760745.377456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825408:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760745.377594:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5818, 5818] 00000080:00200000:1.0:1789760745.377612:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760745.377618:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760745.377635:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23834624 i_blocks 47111 00000020:00200000:1.0:1789760745.377639:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760745.377648:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760745.377683:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23834624, count: 4096 00000080:00200000:1.0:1789760745.377691:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23834624, bytes: 4096 00000080:00200000:1.0:1789760745.377701:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760745.377709:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760745.377711:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23834624, 23838720) 0 0 00000080:00200000:1.0:1789760745.377715:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760745.377733:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5819, 5819] 00020000:00200000:1.0:1789760745.377741:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23834624 chunk: [23834624, 23838720) 23838720, 4096 00020000:00200000:1.0:1789760745.377746:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760745.377765:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5819, 5819] 00000008:00010000:1.0:1789760745.377783:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760745.377789:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760745.377800:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23834624, 23838720) append 0 append lockpos 0 00000080:00200000:1.0:1789760745.377815:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23834624=0x16bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760745.377874:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760745.377876:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23838720 00000020:00010000:1.0:1789760745.377886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760745.377889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.377893:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.377896:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760745.377911:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760745.377920:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760745.377927:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760745.377942:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760745.378074:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 23834624<>23838720 00000100:00100000:0.0:1789760745.378349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495825536:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760745.445330:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760745.445344:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.445355:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.445588:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760745.445620:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495825536:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.445743:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5819, 5819] 00000080:00200000:2.0:1789760745.445873:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.445879:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.445893:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23838720 i_blocks 47111 00000020:00200000:2.0:1789760745.445897:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.445972:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.446005:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23838720, count: 4096 00000080:00200000:2.0:1789760745.446012:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23838720, bytes: 4096 00000080:00200000:2.0:1789760745.446022:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.446030:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.446032:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23838720, 23842816) 0 0 00000080:00200000:2.0:1789760745.446037:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.446053:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5820, 5820] 00020000:00200000:2.0:1789760745.446061:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23838720 chunk: [23838720, 23842816) 23842816, 4096 00020000:00200000:2.0:1789760745.446066:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.446087:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5820, 5820] 00000008:00010000:2.0:1789760745.446106:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.446112:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.446123:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23838720, 23842816) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.446137:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23838720=0x16bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.446224:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.446227:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23842816 00000020:00010000:2.0:1789760745.446234:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.446238:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.446241:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.446244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.446247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.446255:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.446261:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.446276:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760745.446396:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 23838720<>23842816 00000100:00100000:3.0:1789760745.446475:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495825664:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.490733:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973137 gen 2 00000008:00010000:3.0:1789760745.490799:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760745.490808:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.490819:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.490911:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760745.490953:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495825664:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.491100:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5820, 5820] 00000080:00200000:2.0:1789760745.491118:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.491125:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.491142:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23842816 i_blocks 47111 00000020:00200000:2.0:1789760745.491148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.491158:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.491588:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23842816, count: 4096 00000080:00200000:2.0:1789760745.491599:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23842816, bytes: 4096 00000080:00200000:2.0:1789760745.491613:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.491622:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.491625:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23842816, 23846912) 0 0 00000080:00200000:2.0:1789760745.491630:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.491649:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5821, 5821] 00020000:00200000:2.0:1789760745.491658:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23842816 chunk: [23842816, 23846912) 23846912, 4096 00020000:00200000:2.0:1789760745.491663:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.491683:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5821, 5821] 00000008:00010000:2.0:1789760745.491703:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.491708:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.491720:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23842816, 23846912) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.491737:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23842816=0x16bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.491809:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.491812:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23846912 00000020:00010000:2.0:1789760745.491820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.491824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.491827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.491830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.491834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.491842:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.491848:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.491869:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760745.492073:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 23842816<>23846912 00000100:00100000:3.0:1789760745.492160:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825792:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760745.538468:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760745.538485:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760745.538496:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760745.538600:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760745.538637:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495825792:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760745.541846:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5821, 5821] 00000080:00200000:1.0:1789760745.541881:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760745.541888:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760745.541979:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23846912 i_blocks 47111 00000020:00200000:1.0:1789760745.541986:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760745.542005:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760745.542043:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23846912, count: 4096 00000080:00200000:1.0:1789760745.542053:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23846912, bytes: 4096 00000080:00200000:1.0:1789760745.542068:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760745.542077:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760745.542079:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23846912, 23851008) 0 0 00000080:00200000:1.0:1789760745.542084:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760745.542100:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5822, 5822] 00020000:00200000:1.0:1789760745.542109:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23846912 chunk: [23846912, 23851008) 23851008, 4096 00020000:00200000:1.0:1789760745.542114:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760745.542137:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5822, 5822] 00000008:00010000:1.0:1789760745.542155:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760745.542161:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760745.542191:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23846912, 23851008) append 0 append lockpos 0 00000080:00200000:1.0:1789760745.542207:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23846912=0x16be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760745.542400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760745.542404:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23851008 00000020:00010000:1.0:1789760745.542414:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760745.542418:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.542422:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.542426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760745.542429:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760745.542439:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760745.542447:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760745.542478:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760745.542690:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 23846912<>23851008 00000100:00100000:0.0:1789760745.542787:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495825920:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760745.599766:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973139 gen 2 00000008:00010000:1.0:1789760745.599834:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760745.599842:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.599853:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.599876:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760745.599907:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495825920:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.600042:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5822, 5822] 00000080:00200000:2.0:1789760745.600059:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.600066:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.600082:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23851008 i_blocks 47111 00000020:00200000:2.0:1789760745.600086:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.600097:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.600130:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23851008, count: 4096 00000080:00200000:2.0:1789760745.600138:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23851008, bytes: 4096 00000080:00200000:2.0:1789760745.600150:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.600160:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.600163:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23851008, 23855104) 0 0 00000080:00200000:2.0:1789760745.600192:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.600209:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5823, 5823] 00020000:00200000:2.0:1789760745.600219:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23851008 chunk: [23851008, 23855104) 23855104, 4096 00020000:00200000:2.0:1789760745.600224:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.600244:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5823, 5823] 00000008:00010000:2.0:1789760745.600264:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.600270:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.600283:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23851008, 23855104) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.600297:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23851008=0x16bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.600349:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.600352:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23855104 00000020:00010000:2.0:1789760745.600360:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.600364:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.600367:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.600371:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.600373:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.600381:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.600388:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.600403:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760745.600527:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 23851008<>23855104 00000100:00100000:3.0:1789760745.602019:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826048:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760745.657195:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760745.657213:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.657228:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.657260:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760745.657296:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826048:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.660730:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5823, 5823] 00000080:00200000:2.0:1789760745.660770:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.660778:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.660797:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23855104 i_blocks 47111 00000020:00200000:2.0:1789760745.660803:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.661080:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.661118:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23855104, count: 4096 00000080:00200000:2.0:1789760745.661127:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23855104, bytes: 4096 00000080:00200000:2.0:1789760745.661142:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.661151:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.661153:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23855104, 23859200) 0 0 00000080:00200000:2.0:1789760745.661158:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.661215:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5824, 5824] 00020000:00200000:2.0:1789760745.661224:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23855104 chunk: [23855104, 23859200) 23859200, 4096 00020000:00200000:2.0:1789760745.661230:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.661252:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5824, 5824] 00000008:00010000:2.0:1789760745.661269:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.661275:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.661287:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23855104, 23859200) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.661301:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23855104=0x16c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.661364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.661367:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23859200 00000020:00010000:2.0:1789760745.661375:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.661379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.661381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.661384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.661387:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.661394:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.661401:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.661425:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760745.661588:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 23855104<>23859200 00000100:00100000:3.0:1789760745.662334:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760745.662362:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826176:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.710472:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973140 gen 2 00000100:00100000:3.0:1789760745.710531:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205267100 x1876699495826176/t4294973141(4294973141) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760761 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760745.710594:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760745.710601:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.710612:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.710698:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760745.710723:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826176:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.711397:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5824, 5824] 00000080:00200000:2.0:1789760745.711415:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.711422:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.711438:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23859200 i_blocks 47111 00000020:00200000:2.0:1789760745.711442:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.711452:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.711486:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23859200, count: 4096 00000080:00200000:2.0:1789760745.711493:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23859200, bytes: 4096 00000080:00200000:2.0:1789760745.711505:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.711562:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.711565:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23859200, 23863296) 0 0 00000080:00200000:2.0:1789760745.711570:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.711598:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5825, 5825] 00020000:00200000:2.0:1789760745.711607:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23859200 chunk: [23859200, 23863296) 23863296, 4096 00020000:00200000:2.0:1789760745.711612:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.711687:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5825, 5825] 00000008:00010000:2.0:1789760745.711761:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.711768:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.711781:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23859200, 23863296) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.711800:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23859200=0x16c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.711944:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.711947:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23863296 00000020:00010000:2.0:1789760745.711956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.711959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.711962:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.711965:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.711968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.711976:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.711983:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.712005:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760745.712618:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23859200<>23863296 00000100:00100000:3.0:1789760745.712911:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826304:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.742566:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973142 gen 2 00000008:00010000:3.0:1789760745.742638:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760745.742646:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.742657:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.742682:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760745.742712:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826304:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.742749:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5825, 5825] 00000080:00200000:2.0:1789760745.742767:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.742774:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.742791:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23863296 i_blocks 47111 00000020:00200000:2.0:1789760745.742794:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.742804:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.742850:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23863296, count: 4096 00000080:00200000:2.0:1789760745.742857:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23863296, bytes: 4096 00000080:00200000:2.0:1789760745.742866:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.742875:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.742877:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23863296, 23867392) 0 0 00000080:00200000:2.0:1789760745.742882:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.742899:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5826, 5826] 00020000:00200000:2.0:1789760745.742907:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23863296 chunk: [23863296, 23867392) 23867392, 4096 00020000:00200000:2.0:1789760745.742911:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.742931:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5826, 5826] 00000008:00010000:2.0:1789760745.742949:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.742955:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.742966:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23863296, 23867392) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.742980:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23863296=0x16c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.743036:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.743039:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23867392 00000020:00010000:2.0:1789760745.743047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.743050:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.743054:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.743057:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.743060:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.743067:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.743074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.743090:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760745.743257:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 23863296<>23867392 00000100:00100000:3.0:1789760745.743403:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760745.743420:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826432:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.775107:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973143 gen 2 00000008:00010000:3.0:1789760745.775161:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760745.775211:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.775224:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.775247:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760745.775276:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826432:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.775425:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5826, 5826] 00000080:00200000:2.0:1789760745.775443:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.775449:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.775467:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23867392 i_blocks 47111 00000020:00200000:2.0:1789760745.775471:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.775481:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.775517:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23867392, count: 4096 00000080:00200000:2.0:1789760745.775524:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23867392, bytes: 4096 00000080:00200000:2.0:1789760745.775534:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.775547:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.775550:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23867392, 23871488) 0 0 00000080:00200000:2.0:1789760745.775555:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.775589:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5827, 5827] 00020000:00200000:2.0:1789760745.775598:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23867392 chunk: [23867392, 23871488) 23871488, 4096 00020000:00200000:2.0:1789760745.775603:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.775625:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5827, 5827] 00000008:00010000:2.0:1789760745.775645:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.775650:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.775662:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23867392, 23871488) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.775677:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23867392=0x16c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.775733:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.775736:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23871488 00000020:00010000:2.0:1789760745.775744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.775748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.775752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.775756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.775759:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.775766:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.775773:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.775789:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760745.775916:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 23867392<>23871488 00000100:00100000:3.0:1789760745.775986:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760745.811445:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973144 gen 2 00000008:00010000:3.0:1789760745.811506:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760745.811514:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.811525:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.811557:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760745.811595:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495826560:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760745.811677:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5827, 5827] 00000080:00200000:1.0:1789760745.811697:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760745.811703:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760745.811718:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23871488 i_blocks 47111 00000020:00200000:1.0:1789760745.811722:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760745.811733:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760745.811768:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23871488, count: 4096 00000080:00200000:1.0:1789760745.811776:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23871488, bytes: 4096 00000080:00200000:1.0:1789760745.811786:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760745.811795:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760745.811797:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23871488, 23875584) 0 0 00000080:00200000:1.0:1789760745.811801:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760745.811819:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5828, 5828] 00020000:00200000:1.0:1789760745.811827:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23871488 chunk: [23871488, 23875584) 23875584, 4096 00020000:00200000:1.0:1789760745.811833:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760745.811853:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5828, 5828] 00000008:00010000:1.0:1789760745.811873:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760745.811880:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760745.811893:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23871488, 23875584) append 0 append lockpos 0 00000080:00200000:1.0:1789760745.811907:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23871488=0x16c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760745.811969:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760745.811972:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23875584 00000020:00010000:1.0:1789760745.811981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760745.811985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.811988:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.811992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760745.811996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760745.812003:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760745.812011:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760745.812028:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760745.812220:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 23871488<>23875584 00000100:00100000:1.0:1789760745.812257:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760745.812273:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495826688:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760745.846801:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973145 gen 2 00000008:00010000:1.0:1789760745.846856:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760745.846864:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.846875:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.846954:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760745.846984:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495826688:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.847164:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5828, 5828] 00000080:00200000:2.0:1789760745.847208:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.847215:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.847232:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23875584 i_blocks 47111 00000020:00200000:2.0:1789760745.847236:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.847247:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.847287:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23875584, count: 4096 00000080:00200000:2.0:1789760745.847293:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23875584, bytes: 4096 00000080:00200000:2.0:1789760745.847305:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.847314:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.847317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23875584, 23879680) 0 0 00000080:00200000:2.0:1789760745.847322:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.847340:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5829, 5829] 00020000:00200000:2.0:1789760745.847349:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23875584 chunk: [23875584, 23879680) 23879680, 4096 00020000:00200000:2.0:1789760745.847354:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.847573:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5829, 5829] 00000008:00010000:2.0:1789760745.847605:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.847611:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.847624:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23875584, 23879680) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.847639:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23875584=0x16c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.847852:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.847856:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23879680 00000020:00010000:2.0:1789760745.847864:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.847868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.847871:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.847875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.847877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.847885:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.847892:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.847914:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760745.848146:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23875584<>23879680 00000100:00100000:3.0:1789760745.848250:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495826816:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760745.887240:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973146 gen 2 00000008:00010000:2.0:1789760745.887290:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760745.887298:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760745.887310:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760745.887327:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760745.887356:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495826816:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760745.887478:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5829, 5829] 00000080:00200000:1.0:1789760745.887496:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760745.887502:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760745.887518:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23879680 i_blocks 47111 00000020:00200000:1.0:1789760745.887521:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760745.887531:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760745.887568:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23879680, count: 4096 00000080:00200000:1.0:1789760745.887587:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23879680, bytes: 4096 00000080:00200000:1.0:1789760745.887599:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760745.887608:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760745.887610:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23879680, 23883776) 0 0 00000080:00200000:1.0:1789760745.887614:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760745.887632:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5830, 5830] 00020000:00200000:1.0:1789760745.887641:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23879680 chunk: [23879680, 23883776) 23883776, 4096 00020000:00200000:1.0:1789760745.887646:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760745.887667:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5830, 5830] 00000008:00010000:1.0:1789760745.887689:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760745.887695:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760745.887708:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23879680, 23883776) append 0 append lockpos 0 00000080:00200000:1.0:1789760745.887723:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23879680=0x16c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760745.887784:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760745.887787:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23883776 00000020:00010000:1.0:1789760745.887798:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760745.887802:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.887805:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.887808:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760745.887812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760745.887819:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760745.887827:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760745.887843:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760745.887969:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 23879680<>23883776 00000100:00100000:0.0:1789760745.888466:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760745.888489:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495826944:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760745.931745:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760745.934932:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973147 gen 2 00000008:00010000:1.0:1789760745.934983:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760745.934992:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.935002:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.935028:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760745.935057:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495826944:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760745.935232:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5830, 5830] 00000080:00200000:2.0:1789760745.935249:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.935255:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.935271:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23883776 i_blocks 47111 00000020:00200000:2.0:1789760745.935275:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.935285:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.935321:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23883776, count: 4096 00000080:00200000:2.0:1789760745.935328:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23883776, bytes: 4096 00000080:00200000:2.0:1789760745.935338:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.935347:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.935349:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23883776, 23887872) 0 0 00000080:00200000:2.0:1789760745.935353:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.935371:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5831, 5831] 00020000:00200000:2.0:1789760745.935379:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23883776 chunk: [23883776, 23887872) 23887872, 4096 00020000:00200000:2.0:1789760745.935384:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.935404:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5831, 5831] 00000008:00010000:2.0:1789760745.935423:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.935428:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760745.935440:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23883776, 23887872) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.935468:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23883776=0x16c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.935524:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.935528:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23887872 00000020:00010000:2.0:1789760745.935536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.935540:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.935543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.935547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.935550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.935569:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.935584:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.935601:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760745.935720:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23883776<>23887872 00000100:00100000:2.0:1789760745.935769:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760745.935787:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827072:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760745.970330:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973148 gen 2 00000008:00010000:2.0:1789760745.970380:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760745.970388:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760745.970399:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760745.970427:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000080:00200000:1.0:1789760745.970547:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5831, 5831] 00000080:00200000:1.0:1789760745.970631:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760745.970638:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760745.970653:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23887872 i_blocks 47111 00000020:00200000:1.0:1789760745.970657:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760745.970667:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760745.970702:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23887872, count: 4096 00000080:00200000:1.0:1789760745.970711:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23887872, bytes: 4096 00000080:00200000:1.0:1789760745.970723:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760745.970732:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760745.970735:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23887872, 23891968) 0 0 00000080:00200000:1.0:1789760745.970739:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760745.970756:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5832, 5832] 00020000:00200000:1.0:1789760745.970764:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23887872 chunk: [23887872, 23891968) 23891968, 4096 00020000:00200000:1.0:1789760745.970769:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760745.970789:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5832, 5832] 00000008:00010000:1.0:1789760745.970808:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760745.970813:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760745.970825:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23887872, 23891968) append 0 append lockpos 0 00000080:00200000:1.0:1789760745.970838:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23887872=0x16c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760745.970885:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760745.970888:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23891968 00000100:00100000:2.0:1789760745.970973:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827072:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760745.971018:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:2.0:1789760745.971136:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23887872<>23891968 00000100:00100000:2.0:1789760745.971229:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827200:192.168.203.109@tcp:4:dd.0 00000020:00010000:1.0:1789760745.971415:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760745.971419:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.971423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760745.971427:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760745.971429:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760745.971436:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760745.971444:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1789760746.007525:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973149 gen 2 00000008:00010000:2.0:1789760746.007592:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:2.0:1789760746.007600:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760746.007613:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760746.007647:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:2.0:1789760746.007687:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827200:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760746.008809:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5832, 5832] 00000080:00200000:1.0:1789760746.008831:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760746.008837:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760746.008852:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23891968 i_blocks 47111 00000020:00200000:1.0:1789760746.008857:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760746.008867:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760746.008905:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23891968, count: 4096 00000080:00200000:1.0:1789760746.008913:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23891968, bytes: 4096 00000080:00200000:1.0:1789760746.008927:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760746.008936:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760746.008938:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23891968, 23896064) 0 0 00000080:00200000:1.0:1789760746.008942:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760746.008959:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5833, 5833] 00020000:00200000:1.0:1789760746.008968:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23891968 chunk: [23891968, 23896064) 23896064, 4096 00020000:00200000:1.0:1789760746.008973:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760746.008997:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5833, 5833] 00000008:00010000:1.0:1789760746.009018:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760746.009024:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760746.009036:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23891968, 23896064) append 0 append lockpos 0 00000080:00200000:1.0:1789760746.009053:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23891968=0x16c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760746.009328:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760746.009632:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23896064 00000020:00010000:1.0:1789760746.009894:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760746.009898:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.009902:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.009906:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760746.009909:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760746.009920:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760746.009927:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760746.009952:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760746.010094:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 23891968<>23896064 00000100:00100000:0.0:1789760746.010317:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495827328:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760746.069683:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973150 gen 2 00000008:00010000:1.0:1789760746.069758:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760746.069766:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.069778:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.069798:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760746.069822:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495827328:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.070121:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5833, 5833] 00000080:00200000:2.0:1789760746.070138:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.070222:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.070238:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23896064 i_blocks 47111 00000020:00200000:2.0:1789760746.070242:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.070252:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.070284:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23896064, count: 4096 00000080:00200000:2.0:1789760746.070291:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23896064, bytes: 4096 00000080:00200000:2.0:1789760746.070301:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.070310:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.070312:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23896064, 23900160) 0 0 00000080:00200000:2.0:1789760746.070317:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.070333:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5834, 5834] 00020000:00200000:2.0:1789760746.070342:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23896064 chunk: [23896064, 23900160) 23900160, 4096 00020000:00200000:2.0:1789760746.070347:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.070369:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5834, 5834] 00000008:00010000:2.0:1789760746.070388:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.070394:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.070405:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23896064, 23900160) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.070418:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23896064=0x16ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.070469:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.070472:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23900160 00000020:00010000:2.0:1789760746.070480:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.070484:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.070487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.070491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.070494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.070502:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.070508:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.070522:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760746.070721:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 23896064<>23900160 00000100:00100000:3.0:1789760746.070795:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495827456:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.102982:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973151 gen 2 00000008:00010000:3.0:1789760746.103035:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760746.103044:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.103053:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.103080:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760746.103112:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495827456:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.103343:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5834, 5834] 00000080:00200000:2.0:1789760746.103361:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.103367:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.103383:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23900160 i_blocks 47111 00000020:00200000:2.0:1789760746.103387:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.103398:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.103433:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23900160, count: 4096 00000080:00200000:2.0:1789760746.103441:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23900160, bytes: 4096 00000080:00200000:2.0:1789760746.103450:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.103459:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.103461:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23900160, 23904256) 0 0 00000080:00200000:2.0:1789760746.103466:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.103485:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5835, 5835] 00020000:00200000:2.0:1789760746.103493:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23900160 chunk: [23900160, 23904256) 23904256, 4096 00020000:00200000:2.0:1789760746.103498:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.103518:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5835, 5835] 00000008:00010000:2.0:1789760746.103536:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.103542:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.103554:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23900160, 23904256) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.103568:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23900160=0x16cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.103638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.103642:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23904256 00000020:00010000:2.0:1789760746.103650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.103654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.103657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.103660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.103663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.103672:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.103680:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.103697:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760746.103819:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 23900160<>23904256 00000100:00100000:3.0:1789760746.103935:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827584:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.149379:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973152 gen 2 00000008:00010000:3.0:1789760746.149430:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760746.149440:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.149452:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.149528:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760746.149571:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827584:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.149834:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5835, 5835] 00000080:00200000:2.0:1789760746.149852:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.149859:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.149877:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23904256 i_blocks 47111 00000020:00200000:2.0:1789760746.149880:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.149890:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.149992:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23904256, count: 4096 00000080:00200000:2.0:1789760746.150000:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23904256, bytes: 4096 00000080:00200000:2.0:1789760746.150010:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.150019:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.150021:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23904256, 23908352) 0 0 00000080:00200000:2.0:1789760746.150026:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.150045:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5836, 5836] 00020000:00200000:2.0:1789760746.150054:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23904256 chunk: [23904256, 23908352) 23908352, 4096 00020000:00200000:2.0:1789760746.150059:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.150079:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5836, 5836] 00000008:00010000:2.0:1789760746.150395:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.150402:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.150416:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23904256, 23908352) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.150431:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23904256=0x16cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.150496:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.150500:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23908352 00000020:00010000:2.0:1789760746.150509:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.150512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.150515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.150518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.150521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.150528:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.150535:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.150556:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760746.151111:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 23904256<>23908352 00000100:00100000:3.0:1789760746.151264:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495827712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.210071:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973153 gen 2 00000008:00010000:3.0:1789760746.210124:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760746.210132:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.210143:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.210201:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760746.210230:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495827712:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.210333:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5836, 5836] 00000080:00200000:2.0:1789760746.210349:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.210356:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.210372:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23908352 i_blocks 47111 00000020:00200000:2.0:1789760746.210375:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.210384:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.210424:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23908352, count: 4096 00000080:00200000:2.0:1789760746.210432:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23908352, bytes: 4096 00000080:00200000:2.0:1789760746.210441:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.210449:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.210451:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23908352, 23912448) 0 0 00000080:00200000:2.0:1789760746.210455:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.210472:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5837, 5837] 00020000:00200000:2.0:1789760746.210480:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23908352 chunk: [23908352, 23912448) 23912448, 4096 00020000:00200000:2.0:1789760746.210485:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.210503:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5837, 5837] 00000008:00010000:2.0:1789760746.210520:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.210526:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.210538:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23908352, 23912448) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.210550:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23908352=0x16cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.210624:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.210627:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23912448 00000020:00010000:2.0:1789760746.210634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.210637:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.210640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.210643:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.210646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.210652:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.210659:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.210674:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760746.210797:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 23908352<>23912448 00000100:00100000:3.0:1789760746.211021:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827840:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.265065:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973154 gen 2 00000008:00010000:3.0:1789760746.265125:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760746.265135:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.265150:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.265193:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760746.265225:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827840:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.265636:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5837, 5837] 00000080:00200000:2.0:1789760746.265653:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.265659:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.265677:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23912448 i_blocks 47111 00000020:00200000:2.0:1789760746.265681:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.265691:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.265727:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23912448, count: 4096 00000080:00200000:2.0:1789760746.265734:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23912448, bytes: 4096 00000080:00200000:2.0:1789760746.265748:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.265757:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.265760:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23912448, 23916544) 0 0 00000080:00200000:2.0:1789760746.265764:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.265783:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5838, 5838] 00020000:00200000:2.0:1789760746.265791:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23912448 chunk: [23912448, 23916544) 23916544, 4096 00020000:00200000:2.0:1789760746.265796:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.265818:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5838, 5838] 00000008:00010000:2.0:1789760746.265836:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.265842:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.265854:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23912448, 23916544) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.265871:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23912448=0x16ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.265933:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.265936:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23916544 00000020:00010000:2.0:1789760746.265945:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.265949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.265952:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.265956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.265959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.265966:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.266090:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.266110:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760746.266862:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 23912448<>23916544 00000100:00100000:3.0:1789760746.267193:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760746.267219:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827968:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.308746:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973155 gen 2 00000008:00010000:3.0:1789760746.308800:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760746.308808:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.308819:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.308843:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760746.308882:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495827968:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.308914:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5838, 5838] 00000080:00200000:2.0:1789760746.308930:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.308937:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.308953:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23916544 i_blocks 47111 00000020:00200000:2.0:1789760746.308956:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.308966:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.309011:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23916544, count: 4096 00000080:00200000:2.0:1789760746.309018:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23916544, bytes: 4096 00000080:00200000:2.0:1789760746.309027:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.309036:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.309039:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23916544, 23920640) 0 0 00000080:00200000:2.0:1789760746.309043:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.309060:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5839, 5839] 00020000:00200000:2.0:1789760746.309067:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23916544 chunk: [23916544, 23920640) 23920640, 4096 00020000:00200000:2.0:1789760746.309072:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.309094:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5839, 5839] 00000008:00010000:2.0:1789760746.309113:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.309120:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.309133:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23916544, 23920640) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.309148:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23916544=0x16cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.309250:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.309254:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23920640 00000020:00010000:2.0:1789760746.309263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.309267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.309269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.309273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.309275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.309283:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.309289:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.309306:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760746.309434:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 23916544<>23920640 00000100:00100000:3.0:1789760746.309771:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495828096:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.346406:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973156 gen 2 00000008:00010000:3.0:1789760746.346457:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760746.346467:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.346477:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.346503:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760746.346534:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495828096:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.347256:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5839, 5839] 00000080:00200000:2.0:1789760746.347275:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.347282:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.347299:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23920640 i_blocks 47111 00000020:00200000:2.0:1789760746.347303:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.347313:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.347346:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23920640, count: 4096 00000080:00200000:2.0:1789760746.347353:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23920640, bytes: 4096 00000080:00200000:2.0:1789760746.347366:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.347376:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.347379:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23920640, 23924736) 0 0 00000080:00200000:2.0:1789760746.347383:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.347407:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5840, 5840] 00020000:00200000:2.0:1789760746.347416:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23920640 chunk: [23920640, 23924736) 23924736, 4096 00020000:00200000:2.0:1789760746.347420:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.347441:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5840, 5840] 00000008:00010000:2.0:1789760746.347460:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.347465:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.347476:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23920640, 23924736) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.347489:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23920640=0x16d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.347544:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.347546:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23924736 00000020:00010000:2.0:1789760746.347555:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.347559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.347562:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.347565:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.347568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.347585:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.347595:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.347613:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760746.347747:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 23920640<>23924736 00000100:00100000:3.0:1789760746.347826:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495828224:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.387788:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973157 gen 2 00000008:00010000:3.0:1789760746.387836:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760746.387843:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.387854:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.387876:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760746.387911:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495828224:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.388201:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5840, 5840] 00000080:00200000:2.0:1789760746.388220:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.388226:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.388243:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23924736 i_blocks 47111 00000020:00200000:2.0:1789760746.388246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.388256:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.388291:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23924736, count: 4096 00000080:00200000:2.0:1789760746.388298:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23924736, bytes: 4096 00000080:00200000:2.0:1789760746.388308:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.388317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.388320:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23924736, 23928832) 0 0 00000080:00200000:2.0:1789760746.388324:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.388341:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5841, 5841] 00020000:00200000:2.0:1789760746.388350:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23924736 chunk: [23924736, 23928832) 23928832, 4096 00020000:00200000:2.0:1789760746.388355:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.388377:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5841, 5841] 00000008:00010000:2.0:1789760746.388396:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.388402:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.388414:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23924736, 23928832) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.388428:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23924736=0x16d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.388484:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.388486:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23928832 00000020:00010000:2.0:1789760746.388494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.388498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.388502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.388506:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.388509:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.388516:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.388523:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.388541:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760746.388680:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 23924736<>23928832 00000100:00100000:3.0:1789760746.388854:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495828352:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760746.440279:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760746.440297:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.440311:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.440339:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760746.440370:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495828352:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.440480:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5841, 5841] 00000080:00200000:2.0:1789760746.440497:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.440503:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.440518:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23928832 i_blocks 47111 00000020:00200000:2.0:1789760746.440521:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.440530:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.440640:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23928832, count: 4096 00000080:00200000:2.0:1789760746.440649:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23928832, bytes: 4096 00000080:00200000:2.0:1789760746.440661:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.440669:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.440671:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23928832, 23932928) 0 0 00000080:00200000:2.0:1789760746.440676:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.440695:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5842, 5842] 00020000:00200000:2.0:1789760746.440704:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23928832 chunk: [23928832, 23932928) 23932928, 4096 00020000:00200000:2.0:1789760746.440709:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.440729:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5842, 5842] 00000008:00010000:2.0:1789760746.440747:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.440752:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.440763:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23928832, 23932928) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.440779:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23928832=0x16d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.440839:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.440842:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23932928 00000020:00010000:2.0:1789760746.440861:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.440865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.440868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.440871:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.440874:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.440882:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.440888:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.440974:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760746.441108:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 23928832<>23932928 00000100:00100000:3.0:1789760746.441401:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760746.441424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495828480:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.482504:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973158 gen 2 00000100:00100000:3.0:1789760746.482547:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264e00 x1876699495828480/t4294973159(4294973159) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760762 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760746.482813:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760746.482822:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760746.483502:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760746.483706:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760746.483741:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495828480:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760746.483822:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5842, 5842] 00000080:00200000:1.0:1789760746.484000:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760746.484008:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760746.484026:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23932928 i_blocks 47111 00000020:00200000:1.0:1789760746.484030:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760746.484039:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760746.484076:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23932928, count: 4096 00000080:00200000:1.0:1789760746.484083:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23932928, bytes: 4096 00000080:00200000:1.0:1789760746.484096:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760746.484105:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760746.484108:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23932928, 23937024) 0 0 00000080:00200000:1.0:1789760746.484112:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760746.484129:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5843, 5843] 00020000:00200000:1.0:1789760746.484137:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23932928 chunk: [23932928, 23937024) 23937024, 4096 00020000:00200000:1.0:1789760746.484142:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760746.484165:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5843, 5843] 00000008:00010000:1.0:1789760746.484206:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760746.484212:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760746.484225:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23932928, 23937024) append 0 append lockpos 0 00000080:00200000:1.0:1789760746.484242:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23932928=0x16d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760746.484300:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760746.484303:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23937024 00000020:00010000:1.0:1789760746.484311:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760746.484314:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.484317:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.484322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760746.484325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760746.484332:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760746.484338:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760746.484355:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760746.484488:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 23932928<>23937024 00000100:00100000:1.0:1789760746.484540:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760746.484641:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495828608:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760746.542990:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973160 gen 2 00000008:00010000:1.0:1789760746.543055:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760746.543064:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.543074:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.543151:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760746.543223:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495828608:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.543755:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5843, 5843] 00000080:00200000:2.0:1789760746.543773:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.543780:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.543797:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23937024 i_blocks 47111 00000020:00200000:2.0:1789760746.543802:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.543812:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.543847:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23937024, count: 4096 00000080:00200000:2.0:1789760746.543957:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23937024, bytes: 4096 00000080:00200000:2.0:1789760746.543970:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.543979:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.543981:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23937024, 23941120) 0 0 00000080:00200000:2.0:1789760746.543986:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.544004:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5844, 5844] 00020000:00200000:2.0:1789760746.544013:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23937024 chunk: [23937024, 23941120) 23941120, 4096 00020000:00200000:2.0:1789760746.544018:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.544039:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5844, 5844] 00000008:00010000:2.0:1789760746.544059:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.544064:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.544076:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23937024, 23941120) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.544090:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23937024=0x16d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.544152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.544155:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23941120 00000020:00010000:2.0:1789760746.544164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.544188:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.544192:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.544196:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.544199:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.544207:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.544214:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.544235:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760746.544360:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 23937024<>23941120 00000100:00100000:3.0:1789760746.544445:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495828736:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760746.585310:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760746.585325:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.585336:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.585408:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760746.585438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495828736:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760746.585506:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5844, 5844] 00000080:00200000:1.0:1789760746.585522:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760746.585529:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760746.585543:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23941120 i_blocks 47111 00000020:00200000:1.0:1789760746.585547:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760746.585630:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760746.585664:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23941120, count: 4096 00000080:00200000:1.0:1789760746.585671:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23941120, bytes: 4096 00000080:00200000:1.0:1789760746.585681:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760746.585689:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760746.585692:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23941120, 23945216) 0 0 00000080:00200000:1.0:1789760746.585696:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760746.585713:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5845, 5845] 00020000:00200000:1.0:1789760746.585721:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23941120 chunk: [23941120, 23945216) 23945216, 4096 00020000:00200000:1.0:1789760746.585725:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760746.585743:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5845, 5845] 00000008:00010000:1.0:1789760746.585761:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760746.585766:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760746.585777:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23941120, 23945216) append 0 append lockpos 0 00000080:00200000:1.0:1789760746.585790:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23941120=0x16d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760746.585841:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760746.585844:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23945216 00000020:00010000:1.0:1789760746.585852:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760746.585855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.585858:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.585862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760746.585865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760746.585872:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760746.585878:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760746.586491:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760746.586683:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 23941120<>23945216 00000100:00100000:0.0:1789760746.586786:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495828864:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760746.651209:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973162 gen 2 00000008:00010000:1.0:1789760746.651276:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760746.651285:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.651295:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.651325:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760746.651360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495828864:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.651557:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5845, 5845] 00000080:00200000:2.0:1789760746.651582:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.651589:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.651604:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23945216 i_blocks 47111 00000020:00200000:2.0:1789760746.651608:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.651616:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.651651:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23945216, count: 4096 00000080:00200000:2.0:1789760746.651659:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23945216, bytes: 4096 00000080:00200000:2.0:1789760746.651675:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.651683:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.651686:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23945216, 23949312) 0 0 00000080:00200000:2.0:1789760746.651691:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.651709:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5846, 5846] 00020000:00200000:2.0:1789760746.651719:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23945216 chunk: [23945216, 23949312) 23949312, 4096 00020000:00200000:2.0:1789760746.651724:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.651746:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5846, 5846] 00000008:00010000:2.0:1789760746.651767:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.651772:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.651785:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23945216, 23949312) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.651799:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23945216=0x16d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.651858:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.651860:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23949312 00000020:00010000:2.0:1789760746.651869:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.651872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.651876:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.651885:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.651888:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.651895:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.651903:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.651918:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760746.652038:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 23945216<>23949312 00000100:00100000:3.0:1789760746.652438:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495828992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.718995:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973163 gen 2 00000008:00010000:3.0:1789760746.719046:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760746.719055:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.719066:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.719093:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760746.719134:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495828992:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.720522:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5846, 5846] 00000080:00200000:2.0:1789760746.720548:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.720555:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.720585:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23949312 i_blocks 47111 00000020:00200000:2.0:1789760746.720590:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.720604:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.720645:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23949312, count: 4096 00000080:00200000:2.0:1789760746.720654:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23949312, bytes: 4096 00000080:00200000:2.0:1789760746.720680:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.720690:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.720693:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23949312, 23953408) 0 0 00000080:00200000:2.0:1789760746.720698:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.720715:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5847, 5847] 00020000:00200000:2.0:1789760746.720723:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23949312 chunk: [23949312, 23953408) 23953408, 4096 00020000:00200000:2.0:1789760746.720728:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.720750:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5847, 5847] 00000008:00010000:2.0:1789760746.720767:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.720772:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.720784:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23949312, 23953408) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.720798:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23949312=0x16d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.720866:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.720868:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23953408 00000020:00010000:2.0:1789760746.720875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.720879:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.720883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.720886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.720889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.720896:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.720902:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.720921:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760746.721055:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 23949312<>23953408 00000100:00100000:3.0:1789760746.721453:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760746.721480:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.769357:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973164 gen 2 00000008:00010000:3.0:1789760746.769412:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760746.769419:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.769431:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.769453:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760746.769479:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829120:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.769741:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5847, 5847] 00000080:00200000:2.0:1789760746.769758:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.769764:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.769780:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23953408 i_blocks 47111 00000020:00200000:2.0:1789760746.769783:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.769793:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.769831:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23953408, count: 4096 00000080:00200000:2.0:1789760746.769839:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23953408, bytes: 4096 00000080:00200000:2.0:1789760746.769851:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.769860:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.769863:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23953408, 23957504) 0 0 00000080:00200000:2.0:1789760746.769868:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.769884:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5848, 5848] 00020000:00200000:2.0:1789760746.769893:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23953408 chunk: [23953408, 23957504) 23957504, 4096 00020000:00200000:2.0:1789760746.769898:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.769916:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5848, 5848] 00000008:00010000:2.0:1789760746.769932:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.769937:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.769947:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23953408, 23957504) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.769961:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23953408=0x16d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.770011:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.770014:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23957504 00000020:00010000:2.0:1789760746.770021:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.770024:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.770028:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.770032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.770035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.770042:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.770047:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.770062:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760746.770489:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 23953408<>23957504 00000100:00100000:2.0:1789760746.770538:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829248:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760746.814885:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973165 gen 2 00000008:00010000:2.0:1789760746.814934:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760746.814943:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760746.814953:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760746.815022:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760746.815061:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829248:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760746.815708:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5848, 5848] 00000080:00200000:3.0:1789760746.815754:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.815760:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.815964:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23957504 i_blocks 47111 00000020:00200000:2.0:1789760746.815970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.815979:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.816013:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23957504, count: 4096 00000080:00200000:2.0:1789760746.816020:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23957504, bytes: 4096 00000080:00200000:2.0:1789760746.816033:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.816042:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.816044:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23957504, 23961600) 0 0 00000080:00200000:2.0:1789760746.816277:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.816298:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5849, 5849] 00020000:00200000:2.0:1789760746.816306:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23957504 chunk: [23957504, 23961600) 23961600, 4096 00020000:00200000:2.0:1789760746.816311:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.816334:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5849, 5849] 00000008:00010000:2.0:1789760746.816354:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.816360:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.816372:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23957504, 23961600) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.816387:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23957504=0x16d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.816449:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.816452:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23961600 00000020:00010000:2.0:1789760746.816461:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.816464:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.816468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.816471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.816474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.816481:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.816488:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.816534:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760746.816818:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 23957504<>23961600 00000100:00100000:2.0:1789760746.816871:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760746.817402:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829376:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760746.858232:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973166 gen 2 00000008:00010000:2.0:1789760746.858287:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760746.858295:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760746.858309:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760746.858380:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760746.858413:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829376:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760746.858509:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5849, 5849] 00000080:00200000:3.0:1789760746.858557:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.858564:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.858738:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23961600 i_blocks 47111 00000020:00200000:2.0:1789760746.858743:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.858752:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.858790:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23961600, count: 4096 00000080:00200000:2.0:1789760746.858798:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23961600, bytes: 4096 00000080:00200000:2.0:1789760746.858809:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.858818:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.858820:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23961600, 23965696) 0 0 00000080:00200000:2.0:1789760746.858824:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.858842:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5850, 5850] 00020000:00200000:2.0:1789760746.858860:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23961600 chunk: [23961600, 23965696) 23965696, 4096 00020000:00200000:2.0:1789760746.858865:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.859034:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5850, 5850] 00000008:00010000:2.0:1789760746.859056:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.859061:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.859074:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23961600, 23965696) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.859088:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23961600=0x16da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.859224:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.859227:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23965696 00000020:00010000:2.0:1789760746.859236:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.859240:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.859244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.859248:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.859251:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.859260:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.859268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.859315:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760746.859438:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 23961600<>23965696 00000100:00100000:3.0:1789760746.859777:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829504:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760746.905258:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973167 gen 2 00000008:00010000:3.0:1789760746.905319:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760746.905327:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.905339:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.905360:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760746.905390:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829504:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.905977:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5850, 5850] 00000080:00200000:2.0:1789760746.905994:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.906001:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.906018:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23965696 i_blocks 47111 00000020:00200000:2.0:1789760746.906022:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.906031:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.906079:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23965696, count: 4096 00000080:00200000:2.0:1789760746.906086:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23965696, bytes: 4096 00000080:00200000:2.0:1789760746.906098:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.906107:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.906110:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23965696, 23969792) 0 0 00000080:00200000:2.0:1789760746.906114:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.906132:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5851, 5851] 00020000:00200000:2.0:1789760746.906141:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23965696 chunk: [23965696, 23969792) 23969792, 4096 00020000:00200000:2.0:1789760746.906146:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.906166:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5851, 5851] 00000008:00010000:2.0:1789760746.906218:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.906225:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.906237:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23965696, 23969792) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.906251:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23965696=0x16db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.906317:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.906320:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23969792 00000020:00010000:2.0:1789760746.906329:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.906332:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.906335:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.906338:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.906341:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.906349:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.906356:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.906373:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760746.906498:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 23965696<>23969792 00000100:00100000:3.0:1789760746.906786:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495829632:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760746.956151:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760746.960451:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760746.960465:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.960476:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.960498:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760746.960527:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495829632:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760746.960692:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5851, 5851] 00000080:00200000:2.0:1789760746.960709:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.960715:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.960730:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23969792 i_blocks 47111 00000020:00200000:2.0:1789760746.960734:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.960743:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.960776:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23969792, count: 4096 00000080:00200000:2.0:1789760746.960784:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23969792, bytes: 4096 00000080:00200000:2.0:1789760746.960793:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.960801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.960804:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23969792, 23973888) 0 0 00000080:00200000:2.0:1789760746.960808:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.960826:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5852, 5852] 00020000:00200000:2.0:1789760746.960834:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23969792 chunk: [23969792, 23973888) 23973888, 4096 00020000:00200000:2.0:1789760746.960839:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.960858:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5852, 5852] 00000008:00010000:2.0:1789760746.960875:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.960881:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760746.960903:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23969792, 23973888) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.960918:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23969792=0x16dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.960980:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.960983:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23973888 00000020:00010000:2.0:1789760746.960992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.960995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.960998:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.961002:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.961006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.961014:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.961021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.961038:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760746.961447:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23969792<>23973888 00000100:00100000:3.0:1789760746.961717:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760747.005288:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973169 gen 2 00000008:00010000:3.0:1789760747.005355:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760747.005365:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.005376:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.005398:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760747.005431:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495829760:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760747.005794:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5852, 5852] 00000080:00200000:2.0:1789760747.005812:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760747.005817:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760747.005834:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23973888 i_blocks 47111 00000020:00200000:2.0:1789760747.005837:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760747.005846:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760747.005957:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23973888, count: 4096 00000080:00200000:2.0:1789760747.005965:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23973888, bytes: 4096 00000080:00200000:2.0:1789760747.005978:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760747.005987:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760747.005989:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23973888, 23977984) 0 0 00000080:00200000:2.0:1789760747.005994:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760747.006013:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5853, 5853] 00020000:00200000:2.0:1789760747.006021:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23973888 chunk: [23973888, 23977984) 23977984, 4096 00020000:00200000:2.0:1789760747.006026:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760747.006046:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5853, 5853] 00000008:00010000:2.0:1789760747.006133:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760747.006138:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760747.006151:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23973888, 23977984) append 0 append lockpos 0 00000080:00200000:2.0:1789760747.006194:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23973888=0x16dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760747.006255:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760747.006258:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23977984 00000020:00010000:2.0:1789760747.006267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760747.006270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.006273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.006277:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760747.006280:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760747.006289:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760747.006296:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760747.006316:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760747.006533:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 23973888<>23977984 00000100:00100000:3.0:1789760747.006612:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495829888:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760747.055997:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760747.056014:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.056026:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.056104:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760747.056136:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495829888:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760747.056289:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5853, 5853] 00000080:00200000:2.0:1789760747.056306:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760747.056312:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760747.056328:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23977984 i_blocks 47111 00000020:00200000:2.0:1789760747.056332:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760747.056342:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760747.056377:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23977984, count: 4096 00000080:00200000:2.0:1789760747.056384:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23977984, bytes: 4096 00000080:00200000:2.0:1789760747.056395:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760747.056404:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760747.056406:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23977984, 23982080) 0 0 00000080:00200000:2.0:1789760747.056411:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760747.056428:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5854, 5854] 00020000:00200000:2.0:1789760747.056436:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23977984 chunk: [23977984, 23982080) 23982080, 4096 00020000:00200000:2.0:1789760747.056441:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760747.056460:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5854, 5854] 00000008:00010000:2.0:1789760747.056479:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760747.056484:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760747.056496:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23977984, 23982080) append 0 append lockpos 0 00000080:00200000:2.0:1789760747.056510:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23977984=0x16de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760747.056634:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760747.056638:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23982080 00000020:00010000:2.0:1789760747.056647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760747.056650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.056654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.056658:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760747.056661:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760747.056669:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760747.056677:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760747.056695:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760747.056822:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 23977984<>23982080 00000100:00100000:3.0:1789760747.056998:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495830016:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760747.085032:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973170 gen 2 00000100:00100000:2.0:1789760747.085071:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265f80 x1876699495830016/t4294973171(4294973171) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760763 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760747.085120:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:2.0:1789760747.085127:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.085138:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.085257:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:2.0:1789760747.085281:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495830016:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760747.085407:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5854, 5854] 00000080:00200000:1.0:1789760747.085424:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760747.085431:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760747.085445:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23982080 i_blocks 47111 00000020:00200000:1.0:1789760747.085449:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760747.085459:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760747.085495:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23982080, count: 4096 00000080:00200000:1.0:1789760747.085502:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23982080, bytes: 4096 00000080:00200000:1.0:1789760747.085513:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760747.085522:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760747.085525:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23982080, 23986176) 0 0 00000080:00200000:1.0:1789760747.085530:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760747.085612:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5855, 5855] 00020000:00200000:1.0:1789760747.085622:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23982080 chunk: [23982080, 23986176) 23986176, 4096 00020000:00200000:1.0:1789760747.085627:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760747.085647:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5855, 5855] 00000008:00010000:1.0:1789760747.085664:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760747.085669:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760747.085681:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23982080, 23986176) append 0 append lockpos 0 00000080:00200000:1.0:1789760747.085695:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23982080=0x16df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760747.085752:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760747.085755:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23986176 00000020:00010000:1.0:1789760747.085764:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760747.085767:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.085770:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.085773:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760747.085776:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760747.085783:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760747.085790:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.085807:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760747.085991:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 23982080<>23986176 00000100:00100000:0.0:1789760747.086068:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495830144:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.147209:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973172 gen 2 00000008:00010000:1.0:1789760747.147277:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760747.147285:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.147297:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.147384:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760747.147416:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495830144:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760747.151050:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5855, 5855] 00000080:00200000:2.0:1789760747.151148:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760747.151155:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760747.151205:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23986176 i_blocks 47111 00000020:00200000:2.0:1789760747.151211:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760747.151229:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760747.151263:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23986176, count: 4096 00000080:00200000:2.0:1789760747.151272:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23986176, bytes: 4096 00000080:00200000:2.0:1789760747.151286:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760747.151295:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760747.151298:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23986176, 23990272) 0 0 00000080:00200000:2.0:1789760747.151302:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760747.151319:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5856, 5856] 00020000:00200000:2.0:1789760747.151327:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23986176 chunk: [23986176, 23990272) 23990272, 4096 00020000:00200000:2.0:1789760747.151331:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760747.151352:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5856, 5856] 00000008:00010000:2.0:1789760747.151370:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760747.151376:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760747.151387:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23986176, 23990272) append 0 append lockpos 0 00000080:00200000:2.0:1789760747.151401:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23986176=0x16e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760747.151469:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760747.151472:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23990272 00000020:00010000:2.0:1789760747.151481:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760747.151485:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.151489:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.151493:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760747.151495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760747.151503:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760747.151510:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760747.151538:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760747.152452:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 23986176<>23990272 00000100:00100000:3.0:1789760747.152522:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760747.155213:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495830272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760747.204709:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973173 gen 2 00000008:00010000:3.0:1789760747.204764:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760747.204772:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.204785:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.204815:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760747.204857:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495830272:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760747.205268:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5856, 5856] 00000080:00200000:1.0:1789760747.205285:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760747.205291:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760747.205307:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23990272 i_blocks 47111 00000020:00200000:1.0:1789760747.205310:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760747.205319:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760747.205352:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23990272, count: 4096 00000080:00200000:1.0:1789760747.205358:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23990272, bytes: 4096 00000080:00200000:1.0:1789760747.205369:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760747.205377:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760747.205379:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23990272, 23994368) 0 0 00000080:00200000:1.0:1789760747.205384:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760747.205400:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5857, 5857] 00020000:00200000:1.0:1789760747.205407:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23990272 chunk: [23990272, 23994368) 23994368, 4096 00020000:00200000:1.0:1789760747.205412:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760747.205429:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5857, 5857] 00000008:00010000:1.0:1789760747.205447:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760747.205453:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760747.205465:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23990272, 23994368) append 0 append lockpos 0 00000080:00200000:1.0:1789760747.205479:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23990272=0x16e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760747.205532:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760747.205535:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23994368 00000020:00010000:1.0:1789760747.205663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:1.0:1789760747.205668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.205672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.205675:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760747.205678:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760747.205687:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760747.205693:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.205711:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760747.205840:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 23990272<>23994368 00000100:00100000:0.0:1789760747.206032:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495830400:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.284045:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973174 gen 2 00000008:00010000:1.0:1789760747.284089:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760747.284098:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.284108:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.284195:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760747.284220:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495830400:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760747.284332:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5857, 5857] 00000080:00200000:2.0:1789760747.284349:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760747.284356:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760747.284371:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23994368 i_blocks 47111 00000020:00200000:2.0:1789760747.284375:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760747.284384:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760747.284418:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23994368, count: 4096 00000080:00200000:2.0:1789760747.284426:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23994368, bytes: 4096 00000080:00200000:2.0:1789760747.284435:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760747.284443:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760747.284446:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23994368, 23998464) 0 0 00000080:00200000:2.0:1789760747.284450:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760747.284466:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5858, 5858] 00020000:00200000:2.0:1789760747.284473:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23994368 chunk: [23994368, 23998464) 23998464, 4096 00020000:00200000:2.0:1789760747.284478:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760747.284496:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5858, 5858] 00000008:00010000:2.0:1789760747.284511:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760747.284517:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760747.284528:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23994368, 23998464) append 0 append lockpos 0 00000080:00200000:2.0:1789760747.284540:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23994368=0x16e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760747.284665:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760747.284668:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23998464 00000020:00010000:2.0:1789760747.284676:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_release():245 00000020:00010000:2.0:1789760747.284680:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.284683:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.284686:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0368/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760747.284689:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760747.284695:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760747.284701:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760747.284716:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760747.285000:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 23994368<>23998464 00000100:00100000:2.0:1789760747.285478:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760747.285498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495830528:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760747.351589:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973175 gen 2 00000008:00010000:2.0:1789760747.351639:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:2.0:1789760747.351647:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.351657:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.351680:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:2.0:1789760747.351705:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495830528:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760747.351876:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5858, 5858] 00000080:00200000:3.0:1789760747.352026:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.352033:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.352048:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23998464 i_blocks 47111 00000020:00200000:3.0:1789760747.352051:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.352059:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.352103:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23998464, count: 4096 00000080:00200000:3.0:1789760747.352111:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23998464, bytes: 4096 00000080:00200000:3.0:1789760747.352259:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.352269:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.352271:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23998464, 24002560) 0 0 00000080:00200000:3.0:1789760747.352276:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.352292:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5859, 5859] 00020000:00200000:3.0:1789760747.352300:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23998464 chunk: [23998464, 24002560) 24002560, 4096 00020000:00200000:3.0:1789760747.352305:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.352377:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5859, 5859] 00000008:00010000:3.0:1789760747.352394:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.352400:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760747.352411:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23998464, 24002560) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.352423:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=23998464=0x16e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.352473:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.352476:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24002560 00000020:00010000:3.0:1789760747.352483:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.352487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.352490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.352493:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.352496:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.352605:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.352613:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.352632:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760747.352747:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 23998464<>24002560 00000100:00100000:2.0:1789760747.352817:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495830656:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760747.385448:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973176 gen 2 00000008:00010000:2.0:1789760747.385503:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760747.385511:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.385522:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.385538:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760747.385578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495830656:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760747.385941:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5859, 5859] 00000080:00200000:3.0:1789760747.385959:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.385965:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.385979:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24002560 i_blocks 47111 00000020:00200000:3.0:1789760747.385983:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.385992:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.386024:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24002560, count: 4096 00000080:00200000:3.0:1789760747.386031:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24002560, bytes: 4096 00000080:00200000:3.0:1789760747.386041:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.386049:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.386051:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24002560, 24006656) 0 0 00000080:00200000:3.0:1789760747.386055:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.386071:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5860, 5860] 00020000:00200000:3.0:1789760747.386079:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24002560 chunk: [24002560, 24006656) 24006656, 4096 00020000:00200000:3.0:1789760747.386083:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.386103:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5860, 5860] 00000008:00010000:3.0:1789760747.386120:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.386136:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760747.386147:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24002560, 24006656) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.386161:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24002560=0x16e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.386242:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.386245:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24006656 00000020:00010000:3.0:1789760747.386253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.386257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.386260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.386264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.386266:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.386273:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.386279:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.386295:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760747.386415:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 24002560<>24006656 00000100:00100000:2.0:1789760747.386588:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495830784:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760747.441994:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973177 gen 2 00000008:00010000:2.0:1789760747.442044:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760747.442052:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.442062:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.442150:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:2.0:1789760747.442198:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495830784:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760747.444713:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5860, 5860] 00000080:00200000:3.0:1789760747.444741:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.444748:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.444769:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24006656 i_blocks 47111 00000020:00200000:3.0:1789760747.444776:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.444792:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.444829:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24006656, count: 4096 00000080:00200000:3.0:1789760747.444837:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24006656, bytes: 4096 00000080:00200000:3.0:1789760747.444853:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.444862:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.444865:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24006656, 24010752) 0 0 00000080:00200000:3.0:1789760747.444870:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.444888:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5861, 5861] 00020000:00200000:3.0:1789760747.444897:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24006656 chunk: [24006656, 24010752) 24010752, 4096 00020000:00200000:3.0:1789760747.444901:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.444925:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5861, 5861] 00000008:00010000:3.0:1789760747.444944:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.444950:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760747.444964:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24006656, 24010752) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.444980:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24006656=0x16e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.445074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.445077:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24010752 00000020:00010000:3.0:1789760747.445086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.445089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.445092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.445095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.445098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.445107:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.445114:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.445150:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760747.445318:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 24006656<>24010752 00000100:00100000:2.0:1789760747.445662:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495830912:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760747.475450:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973178 gen 2 00000008:00010000:2.0:1789760747.475503:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760747.475513:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.475523:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.475555:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760747.475597:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495830912:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760747.476081:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5861, 5861] 00000080:00200000:3.0:1789760747.476099:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.476105:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.476120:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24010752 i_blocks 47111 00000020:00200000:3.0:1789760747.476124:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.476134:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.476259:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24010752, count: 4096 00000080:00200000:3.0:1789760747.476266:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24010752, bytes: 4096 00000080:00200000:3.0:1789760747.476278:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.476287:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.476290:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24010752, 24014848) 0 0 00000080:00200000:3.0:1789760747.476294:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.476312:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5862, 5862] 00020000:00200000:3.0:1789760747.476321:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24010752 chunk: [24010752, 24014848) 24014848, 4096 00020000:00200000:3.0:1789760747.476326:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.476347:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5862, 5862] 00000008:00010000:3.0:1789760747.476364:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.476370:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760747.476381:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24010752, 24014848) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.476461:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24010752=0x16e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.476519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.476522:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24014848 00000020:00010000:3.0:1789760747.476530:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.476534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.476537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.476612:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.476617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.476626:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.476633:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.476801:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760747.477085:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 24010752<>24014848 00000100:00100000:2.0:1789760747.477224:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495831040:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760747.520086:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973179 gen 2 00000008:00010000:2.0:1789760747.520135:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:2.0:1789760747.520143:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.520154:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.520215:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:2.0:1789760747.520246:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495831040:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760747.520352:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5862, 5862] 00000080:00200000:3.0:1789760747.520368:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.520374:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.520389:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24014848 i_blocks 47111 00000020:00200000:3.0:1789760747.520393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.520403:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.520438:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24014848, count: 4096 00000080:00200000:3.0:1789760747.520446:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24014848, bytes: 4096 00000080:00200000:3.0:1789760747.520455:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.520464:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.520466:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24014848, 24018944) 0 0 00000080:00200000:3.0:1789760747.520471:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.520488:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5863, 5863] 00020000:00200000:3.0:1789760747.520496:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24014848 chunk: [24014848, 24018944) 24018944, 4096 00020000:00200000:3.0:1789760747.520501:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.520522:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5863, 5863] 00000008:00010000:3.0:1789760747.520539:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.520555:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760747.520575:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24014848, 24018944) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.520592:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24014848=0x16e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.520648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.520651:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24018944 00000020:00010000:3.0:1789760747.520661:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.520664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.520668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.520672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.520675:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.520683:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.520690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.520707:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760747.520828:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 24014848<>24018944 00000100:00100000:2.0:1789760747.521072:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495831168:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760747.557279:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:2.0:1789760747.557293:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.557305:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.557321:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:2.0:1789760747.557351:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495831168:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760747.557623:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5863, 5863] 00000080:00200000:3.0:1789760747.557641:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.557648:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.557663:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24018944 i_blocks 47111 00000020:00200000:3.0:1789760747.557667:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.557677:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.557711:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24018944, count: 4096 00000080:00200000:3.0:1789760747.557719:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24018944, bytes: 4096 00000080:00200000:3.0:1789760747.557728:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.557737:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.557740:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24018944, 24023040) 0 0 00000080:00200000:3.0:1789760747.557744:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.557760:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5864, 5864] 00020000:00200000:3.0:1789760747.557768:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24018944 chunk: [24018944, 24023040) 24023040, 4096 00020000:00200000:3.0:1789760747.557773:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.557793:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5864, 5864] 00000008:00010000:3.0:1789760747.557811:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.557817:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760747.557830:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24018944, 24023040) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.557844:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24018944=0x16e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.558023:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.558026:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24023040 00000020:00010000:3.0:1789760747.558035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.558038:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.558041:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.558045:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.558048:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.558057:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.558064:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.558081:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760747.558412:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 24018944<>24023040 00000100:00100000:2.0:1789760747.558479:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495831296:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760747.593687:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973181 gen 2 00000008:00010000:2.0:1789760747.593756:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760747.593765:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.593775:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.593805:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760747.593839:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495831296:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760747.594079:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5864, 5864] 00000080:00200000:3.0:1789760747.594097:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.594103:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.594119:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24023040 i_blocks 47111 00000020:00200000:3.0:1789760747.594123:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.594132:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.594211:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24023040, count: 4096 00000080:00200000:3.0:1789760747.594220:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24023040, bytes: 4096 00000080:00200000:3.0:1789760747.594233:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.594242:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.594244:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24023040, 24027136) 0 0 00000080:00200000:3.0:1789760747.594248:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.594264:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5865, 5865] 00020000:00200000:3.0:1789760747.594272:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24023040 chunk: [24023040, 24027136) 24027136, 4096 00020000:00200000:3.0:1789760747.594277:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.594297:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5865, 5865] 00000008:00010000:3.0:1789760747.594315:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.594321:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760747.594334:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24023040, 24027136) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.594347:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24023040=0x16e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.594404:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.594407:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24027136 00000020:00010000:3.0:1789760747.594416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.594420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.594423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.594427:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.594436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.594443:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.594450:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.594466:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760747.594823:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 24023040<>24027136 00000100:00100000:3.0:1789760747.594871:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495831424:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760747.621935:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973182 gen 2 00000008:00010000:2.0:1789760747.621986:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760747.621994:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.622006:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.622034:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:2.0:1789760747.622065:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495831424:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760747.622242:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5865, 5865] 00000080:00200000:0.0:1789760747.622259:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.622265:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.622280:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24027136 i_blocks 47111 00000020:00200000:0.0:1789760747.622284:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.622293:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.622325:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24027136, count: 4096 00000080:00200000:0.0:1789760747.622333:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24027136, bytes: 4096 00000080:00200000:0.0:1789760747.622344:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.622353:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.622356:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24027136, 24031232) 0 0 00000080:00200000:0.0:1789760747.622361:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.622379:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5866, 5866] 00020000:00200000:0.0:1789760747.622388:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24027136 chunk: [24027136, 24031232) 24031232, 4096 00020000:00200000:0.0:1789760747.622392:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.622413:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5866, 5866] 00000008:00010000:0.0:1789760747.622430:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.622436:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.622448:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24027136, 24031232) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.622462:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24027136=0x16ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760747.622519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.622522:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24031232 00000020:00010000:0.0:1789760747.622531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.622534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.622538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.622541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.622544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.622551:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.622557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760747.622585:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760747.622708:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 24027136<>24031232 00000100:00100000:1.0:1789760747.622791:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760747.622816:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495831552:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760747.670213:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760747.670229:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.670241:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.670329:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760747.670361:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495831552:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760747.672009:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5866, 5866] 00000080:00200000:0.0:1789760747.672038:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.672046:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.672066:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24031232 i_blocks 47111 00000020:00200000:0.0:1789760747.672072:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.672369:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.672419:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24031232, count: 4096 00000080:00200000:0.0:1789760747.672428:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24031232, bytes: 4096 00000080:00200000:0.0:1789760747.672443:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.672454:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.672456:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24031232, 24035328) 0 0 00000080:00200000:0.0:1789760747.672461:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.672479:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5867, 5867] 00020000:00200000:0.0:1789760747.672486:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24031232 chunk: [24031232, 24035328) 24035328, 4096 00020000:00200000:0.0:1789760747.672491:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.672510:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5867, 5867] 00000008:00010000:0.0:1789760747.672528:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.672534:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.672547:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24031232, 24035328) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.672573:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24031232=0x16eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760747.672642:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.672645:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24035328 00000020:00010000:0.0:1789760747.672654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.672657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.672660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.672663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.672666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.672674:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.672681:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760747.672720:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760747.672861:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24031232<>24035328 00000100:00100000:1.0:1789760747.673059:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495831680:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.720671:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973184 gen 2 00000008:00010000:1.0:1789760747.720738:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760747.720746:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.720756:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.720778:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760747.720816:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495831680:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760747.720917:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5867, 5867] 00000080:00200000:0.0:1789760747.720935:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.720949:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.720965:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24035328 i_blocks 47111 00000020:00200000:0.0:1789760747.720969:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.720978:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.721021:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24035328, count: 4096 00000080:00200000:0.0:1789760747.721029:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24035328, bytes: 4096 00000080:00200000:0.0:1789760747.721039:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.721047:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.721055:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24035328, 24039424) 0 0 00000080:00200000:0.0:1789760747.721060:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.721077:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5868, 5868] 00020000:00200000:0.0:1789760747.721085:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24035328 chunk: [24035328, 24039424) 24039424, 4096 00020000:00200000:0.0:1789760747.721090:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.721112:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5868, 5868] 00000008:00010000:0.0:1789760747.721129:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.721135:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.721146:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24035328, 24039424) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.721161:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24035328=0x16ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760747.721255:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.721259:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24039424 00000020:00010000:0.0:1789760747.721268:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.721272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.721275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.721279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.721282:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.721289:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.721297:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760747.721312:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760747.722163:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 24035328<>24039424 00000100:00100000:1.0:1789760747.722584:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495831808:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.761535:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973185 gen 2 00000008:00010000:1.0:1789760747.761597:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760747.761606:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.761624:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.761645:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760747.761687:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495831808:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760747.761856:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5868, 5868] 00000080:00200000:0.0:1789760747.761873:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.761879:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.761896:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24039424 i_blocks 47111 00000020:00200000:0.0:1789760747.761900:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.761909:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.761943:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24039424, count: 4096 00000080:00200000:0.0:1789760747.761950:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24039424, bytes: 4096 00000080:00200000:0.0:1789760747.761960:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.761969:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.761971:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24039424, 24043520) 0 0 00000080:00200000:0.0:1789760747.761975:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.761992:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5869, 5869] 00020000:00200000:0.0:1789760747.762000:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24039424 chunk: [24039424, 24043520) 24043520, 4096 00020000:00200000:0.0:1789760747.762005:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.762025:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5869, 5869] 00000008:00010000:0.0:1789760747.762043:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.762048:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.762060:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24039424, 24043520) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.762075:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24039424=0x16ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760747.762135:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.762139:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24043520 00000020:00010000:0.0:1789760747.762148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.762151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.762154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.762158:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.762160:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.762333:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.762341:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760747.762357:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760747.762487:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 24039424<>24043520 00000100:00100000:1.0:1789760747.762552:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495831936:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.803262:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973186 gen 2 00000008:00010000:1.0:1789760747.803313:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760747.803321:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.803332:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.803354:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760747.803383:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495831936:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760747.803494:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5869, 5869] 00000080:00200000:0.0:1789760747.803511:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.803517:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.803533:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24043520 i_blocks 47111 00000020:00200000:0.0:1789760747.803537:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.803558:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.803602:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24043520, count: 4096 00000080:00200000:0.0:1789760747.803609:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24043520, bytes: 4096 00000080:00200000:0.0:1789760747.803620:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.803629:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.803631:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24043520, 24047616) 0 0 00000080:00200000:0.0:1789760747.803636:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.803651:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5870, 5870] 00020000:00200000:0.0:1789760747.803660:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24043520 chunk: [24043520, 24047616) 24047616, 4096 00020000:00200000:0.0:1789760747.803676:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.803699:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5870, 5870] 00000008:00010000:0.0:1789760747.803718:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.803724:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.803737:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24043520, 24047616) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.803753:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24043520=0x16ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760747.803819:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.803822:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24047616 00000020:00010000:0.0:1789760747.803831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.803835:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.803838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.803842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.803845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.803853:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.803860:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760747.803876:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760747.804017:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 24043520<>24047616 00000100:00100000:1.0:1789760747.804209:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832064:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.836426:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973187 gen 2 00000008:00010000:1.0:1789760747.836477:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760747.836486:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.836497:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.836521:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760747.836567:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832064:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760747.836813:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5870, 5870] 00000080:00200000:0.0:1789760747.836831:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.836837:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.836864:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24047616 i_blocks 47111 00000020:00200000:0.0:1789760747.836868:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.836877:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.836921:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24047616, count: 4096 00000080:00200000:0.0:1789760747.836928:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24047616, bytes: 4096 00000080:00200000:0.0:1789760747.836938:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.836946:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.836948:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24047616, 24051712) 0 0 00000080:00200000:0.0:1789760747.836953:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.836970:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5871, 5871] 00020000:00200000:0.0:1789760747.836978:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24047616 chunk: [24047616, 24051712) 24051712, 4096 00020000:00200000:0.0:1789760747.836983:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.837003:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5871, 5871] 00000008:00010000:0.0:1789760747.837021:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.837027:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.837039:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24047616, 24051712) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.837054:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24047616=0x16ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760747.837111:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.837114:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24051712 00000020:00010000:0.0:1789760747.837123:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.837126:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.837129:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.837132:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.837135:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.837142:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.837150:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760747.837164:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760747.837340:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 24047616<>24051712 00000100:00100000:1.0:1789760747.837772:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760747.837792:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832192:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.871437:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973188 gen 2 00000008:00010000:1.0:1789760747.871493:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760747.871505:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.871520:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.871546:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760747.871594:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832192:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760747.871805:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5871, 5871] 00000080:00200000:0.0:1789760747.871827:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.871834:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.871998:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24051712 i_blocks 47111 00000020:00200000:0.0:1789760747.872004:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.872014:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.872057:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24051712, count: 4096 00000080:00200000:0.0:1789760747.872065:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24051712, bytes: 4096 00000080:00200000:0.0:1789760747.872075:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.872085:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.872087:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24051712, 24055808) 0 0 00000080:00200000:0.0:1789760747.872092:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.872110:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5872, 5872] 00020000:00200000:0.0:1789760747.872118:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24051712 chunk: [24051712, 24055808) 24055808, 4096 00020000:00200000:0.0:1789760747.872123:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.872143:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5872, 5872] 00000008:00010000:0.0:1789760747.872160:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.872166:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.872201:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24051712, 24055808) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.872216:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24051712=0x16f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760747.872277:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.872280:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24055808 00000020:00010000:0.0:1789760747.872289:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.872293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.872296:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.872300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.872303:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.872310:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.872317:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760747.872334:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760747.872476:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 24051712<>24055808 00000100:00100000:1.0:1789760747.872605:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832320:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.917006:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973189 gen 2 00000008:00010000:1.0:1789760747.917059:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760747.917068:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.917078:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.917099:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000080:00200000:0.0:1789760747.917190:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5872, 5872] 00000080:00200000:0.0:1789760747.917208:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.917215:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.917231:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24055808 i_blocks 47111 00000020:00200000:0.0:1789760747.917235:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.917244:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.917276:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24055808, count: 4096 00000080:00200000:0.0:1789760747.917285:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24055808, bytes: 4096 00000080:00200000:0.0:1789760747.917295:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.917304:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.917306:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24055808, 24059904) 0 0 00000080:00200000:0.0:1789760747.917311:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.917327:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5873, 5873] 00020000:00200000:0.0:1789760747.917336:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24055808 chunk: [24055808, 24059904) 24059904, 4096 00020000:00200000:0.0:1789760747.917341:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.917371:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5873, 5873] 00000008:00010000:0.0:1789760747.917389:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.917395:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.917406:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24055808, 24059904) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.917421:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24055808=0x16f1000, pages 1 (max 131072), lockless, parallel 00000100:00100000:1.0:1789760747.917516:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832320:192.168.203.109@tcp:4:dd.0 00000020:00200000:0.0:1789760747.919426:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.919439:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24059904 00000020:00010000:0.0:1789760747.919456:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.919462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.919467:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.919471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.919474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.919487:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.919496:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.919701:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:1.0:1789760747.919922:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24055808<>24059904 00000100:00100000:1.0:1789760747.920006:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495832448:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760747.965522:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973190 gen 2 00000008:00010000:1.0:1789760747.965590:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760747.965600:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.965612:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.965633:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760747.965678:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495832448:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760747.965818:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5873, 5873] 00000080:00200000:0.0:1789760747.965835:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760747.965842:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760747.965857:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24059904 i_blocks 47111 00000020:00200000:0.0:1789760747.965861:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760747.965870:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760747.965914:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24059904, count: 4096 00000080:00200000:0.0:1789760747.965922:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24059904, bytes: 4096 00000080:00200000:0.0:1789760747.965931:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760747.965941:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760747.965943:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24059904, 24064000) 0 0 00000080:00200000:0.0:1789760747.965957:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760747.965979:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5874, 5874] 00020000:00200000:0.0:1789760747.965988:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24059904 chunk: [24059904, 24064000) 24064000, 4096 00020000:00200000:0.0:1789760747.965995:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760747.966016:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5874, 5874] 00000008:00010000:0.0:1789760747.966230:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760747.966237:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760747.966250:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24059904, 24064000) append 0 append lockpos 0 00000080:00200000:0.0:1789760747.966264:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24059904=0x16f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760747.966319:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760747.966323:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24064000 00000020:00010000:0.0:1789760747.966331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760747.966334:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.966337:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760747.966341:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760747.966343:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760747.966352:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760747.966359:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.967079:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:1.0:1789760747.967233:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24059904<>24064000 00000100:00100000:1.0:1789760747.967269:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760747.967290:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495832576:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760747.980902:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760748.007103:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973191 gen 2 00000008:00010000:1.0:1789760748.007219:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760748.007226:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.007392:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.007412:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760748.007440:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495832576:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.007589:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5874, 5874] 00000080:00200000:0.0:1789760748.007604:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.007609:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.007622:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24064000 i_blocks 47111 00000020:00200000:0.0:1789760748.007726:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.007734:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.007758:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24064000, count: 4096 00000080:00200000:0.0:1789760748.007766:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24064000, bytes: 4096 00000080:00200000:0.0:1789760748.007774:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.007781:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.007784:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24064000, 24068096) 0 0 00000080:00200000:0.0:1789760748.007788:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.007802:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5875, 5875] 00020000:00200000:0.0:1789760748.007809:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24064000 chunk: [24064000, 24068096) 24068096, 4096 00020000:00200000:0.0:1789760748.007813:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.007828:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5875, 5875] 00000008:00010000:0.0:1789760748.007842:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.007848:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.007874:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24064000, 24068096) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.009201:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24064000=0x16f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.009396:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.009400:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24068096 00000020:00010000:0.0:1789760748.009409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.009413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.009417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.009422:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.009425:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.009435:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.009442:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.009466:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760748.009721:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 24064000<>24068096 00000100:00100000:0.0:1789760748.009772:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760748.009792:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832704:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.061659:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973192 gen 2 00000008:00010000:1.0:1789760748.061698:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760748.061705:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.061714:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.061735:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760748.061759:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832704:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.061905:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5875, 5875] 00000080:00200000:0.0:1789760748.061916:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.061921:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.061933:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24068096 i_blocks 47111 00000020:00200000:0.0:1789760748.061936:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.061944:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.061970:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24068096, count: 4096 00000080:00200000:0.0:1789760748.061976:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24068096, bytes: 4096 00000080:00200000:0.0:1789760748.061983:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.061990:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.061992:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24068096, 24072192) 0 0 00000080:00200000:0.0:1789760748.061995:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.062008:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5876, 5876] 00020000:00200000:0.0:1789760748.062014:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24068096 chunk: [24068096, 24072192) 24072192, 4096 00020000:00200000:0.0:1789760748.062018:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.062033:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5876, 5876] 00000008:00010000:0.0:1789760748.062047:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.062052:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.062061:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24068096, 24072192) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.062072:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24068096=0x16f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.062395:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.062399:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24072192 00000020:00010000:0.0:1789760748.062407:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.062410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.062413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.062416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.062419:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.062425:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.062430:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.062459:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760748.064750:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 24068096<>24072192 00000100:00100000:1.0:1789760748.065269:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832832:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.123141:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973193 gen 2 00000008:00010000:1.0:1789760748.123208:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760748.123217:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.123230:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.123256:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760748.123414:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832832:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.124002:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5876, 5876] 00000080:00200000:0.0:1789760748.124021:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.124028:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.124044:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24072192 i_blocks 47111 00000020:00200000:0.0:1789760748.124048:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.124059:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.124094:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24072192, count: 4096 00000080:00200000:0.0:1789760748.124100:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24072192, bytes: 4096 00000080:00200000:0.0:1789760748.124125:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.124135:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.124138:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24072192, 24076288) 0 0 00000080:00200000:0.0:1789760748.124142:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.124158:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5877, 5877] 00020000:00200000:0.0:1789760748.124207:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24072192 chunk: [24072192, 24076288) 24076288, 4096 00020000:00200000:0.0:1789760748.124212:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.124235:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5877, 5877] 00000008:00010000:0.0:1789760748.124254:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.124260:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.124272:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24072192, 24076288) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.124286:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24072192=0x16f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.124345:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.124348:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24076288 00000020:00010000:0.0:1789760748.124486:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.124491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.124495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.124499:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.124502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.124511:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.124518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.124678:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760748.124811:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 24072192<>24076288 00000100:00100000:1.0:1789760748.125073:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760748.125100:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832960:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.176964:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973194 gen 2 00000008:00010000:1.0:1789760748.177013:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760748.177021:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.177032:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.177052:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760748.177080:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495832960:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.177330:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5877, 5877] 00000080:00200000:0.0:1789760748.177348:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.177355:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.177372:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24076288 i_blocks 47111 00000020:00200000:0.0:1789760748.177376:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.177385:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.177421:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24076288, count: 4096 00000080:00200000:0.0:1789760748.177429:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24076288, bytes: 4096 00000080:00200000:0.0:1789760748.177440:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.177450:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.177452:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24076288, 24080384) 0 0 00000080:00200000:0.0:1789760748.177457:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.177476:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5878, 5878] 00020000:00200000:0.0:1789760748.177485:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24076288 chunk: [24076288, 24080384) 24080384, 4096 00020000:00200000:0.0:1789760748.177489:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.177509:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5878, 5878] 00000008:00010000:0.0:1789760748.178823:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.178834:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.178850:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24076288, 24080384) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.178866:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24076288=0x16f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.179012:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.179016:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24080384 00000020:00010000:0.0:1789760748.179026:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.179030:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.179033:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.179037:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.179039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.179050:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.179058:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.179081:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760748.180790:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 24076288<>24080384 00000100:00100000:1.0:1789760748.181009:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495833088:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.228868:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973195 gen 2 00000008:00010000:1.0:1789760748.228919:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760748.228928:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.228939:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.229106:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760748.229293:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495833088:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.229394:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5878, 5878] 00000080:00200000:0.0:1789760748.229411:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.229420:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.229499:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24080384 i_blocks 47111 00000020:00200000:1.0:1789760748.229503:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.229512:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.229548:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24080384, count: 4096 00000080:00200000:1.0:1789760748.229554:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24080384, bytes: 4096 00000080:00200000:1.0:1789760748.229577:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.229585:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.229588:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24080384, 24084480) 0 0 00000080:00200000:1.0:1789760748.229592:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.229616:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5879, 5879] 00020000:00200000:1.0:1789760748.229624:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24080384 chunk: [24080384, 24084480) 24084480, 4096 00020000:00200000:1.0:1789760748.229629:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.229648:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5879, 5879] 00000008:00010000:1.0:1789760748.229667:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.229672:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760748.229684:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24080384, 24084480) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.229698:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24080384=0x16f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.229750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.229753:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24084480 00000020:00010000:1.0:1789760748.229762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.229765:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.229769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.229772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.229776:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.229784:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.229791:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.229804:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760748.229918:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24080384<>24084480 00000100:00100000:1.0:1789760748.229961:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760748.229977:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495833216:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.266664:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973196 gen 2 00000008:00010000:1.0:1789760748.266716:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760748.266724:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.266734:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.266756:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760748.266788:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495833216:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.266789:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5879, 5879] 00000080:00200000:0.0:1789760748.266811:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.266818:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.266835:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24084480 i_blocks 47111 00000020:00200000:0.0:1789760748.266839:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.266851:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.266900:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24084480, count: 4096 00000080:00200000:0.0:1789760748.266907:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24084480, bytes: 4096 00000080:00200000:0.0:1789760748.266919:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.266928:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.266930:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24084480, 24088576) 0 0 00000080:00200000:0.0:1789760748.266935:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.266953:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5880, 5880] 00020000:00200000:0.0:1789760748.266959:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24084480 chunk: [24084480, 24088576) 24088576, 4096 00020000:00200000:0.0:1789760748.266964:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.266982:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5880, 5880] 00000008:00010000:0.0:1789760748.266998:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.267002:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.267012:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24084480, 24088576) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.267025:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24084480=0x16f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.267099:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.267101:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24088576 00000020:00010000:0.0:1789760748.267109:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.267112:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.267114:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.267117:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.267119:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.267124:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.267129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.267140:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760748.267298:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 24084480<>24088576 00000100:00100000:0.0:1789760748.267331:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760748.267347:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495833344:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.310657:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973197 gen 2 00000008:00010000:1.0:1789760748.310712:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760748.310720:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.310731:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.310799:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760748.310830:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495833344:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.310920:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5880, 5880] 00000080:00200000:0.0:1789760748.310939:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.310946:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.310963:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24088576 i_blocks 47111 00000020:00200000:0.0:1789760748.310966:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.310975:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.311131:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24088576, count: 4096 00000080:00200000:0.0:1789760748.311140:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24088576, bytes: 4096 00000080:00200000:0.0:1789760748.311152:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.311160:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.311162:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24088576, 24092672) 0 0 00000080:00200000:0.0:1789760748.311167:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.311207:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5881, 5881] 00020000:00200000:0.0:1789760748.311216:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24088576 chunk: [24088576, 24092672) 24092672, 4096 00020000:00200000:0.0:1789760748.311221:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.311240:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5881, 5881] 00000008:00010000:0.0:1789760748.311687:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.311695:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.311708:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24088576, 24092672) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.311724:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24088576=0x16f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.311786:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.311789:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24092672 00000020:00010000:0.0:1789760748.311796:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.311812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.311815:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.311819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.311822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.311831:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.311838:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.311857:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760748.311984:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 24088576<>24092672 00000100:00100000:1.0:1789760748.312207:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495833472:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.353067:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973198 gen 2 00000008:00010000:1.0:1789760748.353118:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760748.353127:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.353137:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.353159:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760748.353222:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495833472:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.353330:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5881, 5881] 00000080:00200000:0.0:1789760748.353351:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.353357:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.353374:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24092672 i_blocks 47111 00000020:00200000:0.0:1789760748.353378:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.353386:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.353428:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24092672, count: 4096 00000080:00200000:0.0:1789760748.353436:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24092672, bytes: 4096 00000080:00200000:0.0:1789760748.353446:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.353454:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.353456:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24092672, 24096768) 0 0 00000080:00200000:0.0:1789760748.353460:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.353478:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5882, 5882] 00020000:00200000:0.0:1789760748.353487:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24092672 chunk: [24092672, 24096768) 24096768, 4096 00020000:00200000:0.0:1789760748.353492:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.353511:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5882, 5882] 00000008:00010000:0.0:1789760748.353528:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.353735:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.353750:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24092672, 24096768) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.353764:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24092672=0x16fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.354163:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.354167:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24096768 00000020:00010000:0.0:1789760748.354210:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.354214:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.354217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.354220:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.354223:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.354234:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.354241:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.354259:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760748.354385:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 24092672<>24096768 00000100:00100000:1.0:1789760748.354485:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760748.354506:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495833600:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.403915:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973199 gen 2 00000008:00010000:1.0:1789760748.403967:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760748.403976:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.403987:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.404010:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760748.404038:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495833600:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.404251:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5882, 5882] 00000080:00200000:0.0:1789760748.404268:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.404274:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.404290:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24096768 i_blocks 47111 00000020:00200000:0.0:1789760748.404293:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.404302:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.404335:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24096768, count: 4096 00000080:00200000:0.0:1789760748.404342:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24096768, bytes: 4096 00000080:00200000:0.0:1789760748.404351:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.404361:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.404363:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24096768, 24100864) 0 0 00000080:00200000:0.0:1789760748.404367:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.404385:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5883, 5883] 00020000:00200000:0.0:1789760748.404393:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24096768 chunk: [24096768, 24100864) 24100864, 4096 00020000:00200000:0.0:1789760748.404398:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.404418:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5883, 5883] 00000008:00010000:0.0:1789760748.404436:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.404442:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.404453:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24096768, 24100864) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.404467:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24096768=0x16fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.404520:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.404522:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24100864 00000020:00010000:0.0:1789760748.404531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.404534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.404537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.404541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.404544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.404552:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.404570:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.404584:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760748.404702:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 24096768<>24100864 00000100:00100000:1.0:1789760748.404856:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495833728:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.427607:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495833856:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760748.427727:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495834112:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760748.427770:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760748.427777:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495833984:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760748.433902:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495833984:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760748.433924:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495834112:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760748.437935:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495833856:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760748.446929:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973200 gen 2 00000008:00010000:1.0:1789760748.446978:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760748.446987:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.446997:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.447020:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760748.447048:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495833728:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.447121:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5883, 5883] 00000080:00200000:0.0:1789760748.447139:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.447146:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.447162:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24100864 i_blocks 47111 00000020:00200000:0.0:1789760748.447166:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.447309:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.447344:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24100864, count: 4096 00000080:00200000:0.0:1789760748.447352:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24100864, bytes: 4096 00000080:00200000:0.0:1789760748.447361:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.447369:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.447372:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24100864, 24104960) 0 0 00000080:00200000:0.0:1789760748.447376:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.447392:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5884, 5884] 00020000:00200000:0.0:1789760748.447400:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24100864 chunk: [24100864, 24104960) 24104960, 4096 00020000:00200000:0.0:1789760748.447405:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.447425:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5884, 5884] 00000008:00010000:0.0:1789760748.447442:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.447447:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.447459:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24100864, 24104960) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.447473:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24100864=0x16fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.447527:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.447530:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24104960 00000020:00010000:0.0:1789760748.447549:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.447553:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.447564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.447568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.447571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.447578:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.447586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.447601:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760748.447724:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 24100864<>24104960 00000100:00100000:1.0:1789760748.447920:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495834240:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760748.512039:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973201 gen 2 00000008:00010000:1.0:1789760748.512105:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760748.512114:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.512126:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.512152:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760748.512966:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495834240:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.516441:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5884, 5884] 00000080:00200000:2.0:1789760748.516467:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.516474:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.516493:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24104960 i_blocks 47111 00000020:00200000:2.0:1789760748.516499:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.516515:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.516548:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24104960, count: 4096 00000080:00200000:2.0:1789760748.516565:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24104960, bytes: 4096 00000080:00200000:2.0:1789760748.516580:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.516590:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.516593:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24104960, 24109056) 0 0 00000080:00200000:2.0:1789760748.516597:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.516614:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5885, 5885] 00020000:00200000:2.0:1789760748.516623:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24104960 chunk: [24104960, 24109056) 24109056, 4096 00020000:00200000:2.0:1789760748.516627:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.516649:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5885, 5885] 00000008:00010000:2.0:1789760748.516667:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.516672:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.516684:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24104960, 24109056) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.516698:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24104960=0x16fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.516762:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.516765:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24109056 00000020:00010000:2.0:1789760748.516774:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.516778:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.516781:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.516784:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.516787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.516796:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.516803:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.516824:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760748.516950:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 24104960<>24109056 00000100:00100000:2.0:1789760748.517015:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760748.517038:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495834368:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760748.554796:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:2.0:1789760748.554812:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760748.554824:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760748.555009:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:2.0:1789760748.555059:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495834368:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.555331:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5885, 5885] 00000080:00200000:0.0:1789760748.555347:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.555354:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.555371:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24109056 i_blocks 47111 00000020:00200000:0.0:1789760748.555374:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.555453:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.555489:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24109056, count: 4096 00000080:00200000:0.0:1789760748.555496:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24109056, bytes: 4096 00000080:00200000:0.0:1789760748.555508:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.555517:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.555519:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24109056, 24113152) 0 0 00000080:00200000:0.0:1789760748.555524:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.555541:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5886, 5886] 00020000:00200000:0.0:1789760748.555550:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24109056 chunk: [24109056, 24113152) 24113152, 4096 00020000:00200000:0.0:1789760748.555563:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.555583:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5886, 5886] 00000008:00010000:0.0:1789760748.555600:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.555606:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.555616:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24109056, 24113152) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.555630:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24109056=0x16fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.555686:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.555689:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24113152 00000020:00010000:0.0:1789760748.555697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.555700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.555703:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.555706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.555709:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.555716:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.555722:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.555738:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760748.555859:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 24109056<>24113152 00000100:00100000:1.0:1789760748.555988:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495834496:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760748.587567:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973203 gen 2 00000008:00010000:0.0:1789760748.587633:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760748.587642:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760748.587652:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760748.587752:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760748.587783:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495834496:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.587898:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5886, 5886] 00000080:00200000:2.0:1789760748.587914:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.587922:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.587937:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24113152 i_blocks 47111 00000020:00200000:2.0:1789760748.587941:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.587950:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.587984:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24113152, count: 4096 00000080:00200000:2.0:1789760748.587992:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24113152, bytes: 4096 00000080:00200000:2.0:1789760748.588003:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.588012:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.588015:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24113152, 24117248) 0 0 00000080:00200000:2.0:1789760748.588020:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.588037:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5887, 5887] 00020000:00200000:2.0:1789760748.588046:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24113152 chunk: [24113152, 24117248) 24117248, 4096 00020000:00200000:2.0:1789760748.588052:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.588072:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5887, 5887] 00000008:00010000:2.0:1789760748.588091:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.588097:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.588110:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24113152, 24117248) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.588124:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24113152=0x16ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.589345:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.589354:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24117248 00000020:00010000:2.0:1789760748.589368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.589372:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.589376:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.589379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.589382:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.589393:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.589400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.589429:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760748.589576:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 24113152<>24117248 00000100:00100000:3.0:1789760748.589668:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760748.589695:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495834624:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.625853:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973204 gen 2 00000008:00010000:3.0:1789760748.625898:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760748.625906:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.625916:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.625939:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760748.625969:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495834624:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.626204:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5887, 5887] 00000080:00200000:2.0:1789760748.626221:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.626227:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.626243:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24117248 i_blocks 47111 00000020:00200000:2.0:1789760748.626247:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.626257:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.626292:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24117248, count: 4096 00000080:00200000:2.0:1789760748.626299:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24117248, bytes: 4096 00000080:00200000:2.0:1789760748.626308:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.626317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.626319:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24117248, 24121344) 0 0 00000080:00200000:2.0:1789760748.626324:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.626394:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5888, 5888] 00020000:00200000:2.0:1789760748.626403:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24117248 chunk: [24117248, 24121344) 24121344, 4096 00020000:00200000:2.0:1789760748.626408:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.626430:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5888, 5888] 00000008:00010000:2.0:1789760748.626626:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.626634:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.626656:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24117248, 24121344) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.626672:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24117248=0x1700000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.626738:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.626741:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24121344 00000020:00010000:2.0:1789760748.626750:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.626753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.626756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.626760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.626763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.626771:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.626779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.626799:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760748.626922:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 24117248<>24121344 00000100:00100000:3.0:1789760748.627083:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760748.627105:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495834752:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.676427:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973205 gen 2 00000008:00010000:3.0:1789760748.676480:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760748.676488:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.676500:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.676524:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760748.676564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495834752:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.677200:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5888, 5888] 00000080:00200000:2.0:1789760748.677219:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.677226:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.677245:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24121344 i_blocks 47111 00000020:00200000:2.0:1789760748.677248:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.677258:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.677294:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24121344, count: 4096 00000080:00200000:2.0:1789760748.677301:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24121344, bytes: 4096 00000080:00200000:2.0:1789760748.677312:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.677320:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.677322:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24121344, 24125440) 0 0 00000080:00200000:2.0:1789760748.677345:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.677365:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5889, 5889] 00020000:00200000:2.0:1789760748.677373:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24121344 chunk: [24121344, 24125440) 24125440, 4096 00020000:00200000:2.0:1789760748.677378:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.677398:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5889, 5889] 00000008:00010000:2.0:1789760748.677417:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.677423:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.677436:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24121344, 24125440) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.677451:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24121344=0x1701000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.677511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.677514:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24125440 00000020:00010000:2.0:1789760748.677522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.677526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.677529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.677532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.677535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.677542:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.677550:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.677581:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760748.677708:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 24121344<>24125440 00000100:00100000:3.0:1789760748.677845:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760748.677862:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495834880:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.715073:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973206 gen 2 00000008:00010000:3.0:1789760748.715122:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760748.715130:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.715142:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.715184:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760748.715209:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495834880:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.715313:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5889, 5889] 00000080:00200000:2.0:1789760748.715330:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.715337:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.715353:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24125440 i_blocks 49159 00000020:00200000:2.0:1789760748.715357:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.715366:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.715399:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24125440, count: 4096 00000080:00200000:2.0:1789760748.715406:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24125440, bytes: 4096 00000080:00200000:2.0:1789760748.715415:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.715424:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.715426:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24125440, 24129536) 0 0 00000080:00200000:2.0:1789760748.715431:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.715448:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5890, 5890] 00020000:00200000:2.0:1789760748.715456:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24125440 chunk: [24125440, 24129536) 24129536, 4096 00020000:00200000:2.0:1789760748.715461:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.715481:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5890, 5890] 00000008:00010000:2.0:1789760748.715499:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.715505:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.715516:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24125440, 24129536) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.715531:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24125440=0x1702000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.716713:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.716721:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24129536 00000020:00010000:2.0:1789760748.716733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.716737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.716741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.716745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.716748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.716757:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.716766:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.716789:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760748.716916:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 24125440<>24129536 00000100:00100000:3.0:1789760748.717045:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760748.717070:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495835008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.758624:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973207 gen 2 00000008:00010000:3.0:1789760748.758675:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760748.758684:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.758696:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.758726:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760748.758758:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495835008:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.759009:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5890, 5890] 00000080:00200000:2.0:1789760748.759026:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.759033:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.759048:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24129536 i_blocks 49159 00000020:00200000:2.0:1789760748.759052:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.759061:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.759092:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24129536, count: 4096 00000080:00200000:2.0:1789760748.759098:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24129536, bytes: 4096 00000080:00200000:2.0:1789760748.759110:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.759118:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.759121:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24129536, 24133632) 0 0 00000080:00200000:2.0:1789760748.759126:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.759142:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5891, 5891] 00020000:00200000:2.0:1789760748.759150:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24129536 chunk: [24129536, 24133632) 24133632, 4096 00020000:00200000:2.0:1789760748.759155:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.759206:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5891, 5891] 00000008:00010000:2.0:1789760748.759225:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.759230:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.759241:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24129536, 24133632) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.759255:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24129536=0x1703000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.759827:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.759833:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24133632 00000020:00010000:2.0:1789760748.759842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.759846:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.759849:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.759853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.759856:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.759864:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.759871:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.759901:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760748.760027:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 24129536<>24133632 00000100:00100000:3.0:1789760748.760235:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760748.760257:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495835136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.786704:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973208 gen 2 00000008:00010000:3.0:1789760748.786760:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760748.786769:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.786782:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.786807:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760748.786836:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495835136:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.787065:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5891, 5891] 00000080:00200000:2.0:1789760748.787082:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.787088:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.787103:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24133632 i_blocks 49159 00000020:00200000:2.0:1789760748.787107:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.787116:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.787152:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24133632, count: 4096 00000080:00200000:2.0:1789760748.787159:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24133632, bytes: 4096 00000080:00200000:2.0:1789760748.787217:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.787227:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.787230:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24133632, 24137728) 0 0 00000080:00200000:2.0:1789760748.787235:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.787252:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5892, 5892] 00020000:00200000:2.0:1789760748.787260:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24133632 chunk: [24133632, 24137728) 24137728, 4096 00020000:00200000:2.0:1789760748.787265:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.787287:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5892, 5892] 00000008:00010000:2.0:1789760748.787304:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.787310:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.787321:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24133632, 24137728) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.787334:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24133632=0x1704000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.787387:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.787390:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24137728 00000020:00010000:2.0:1789760748.787398:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.787402:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.787405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.787409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.787412:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.787419:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.787426:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.787443:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760748.787582:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 24133632<>24137728 00000100:00100000:3.0:1789760748.787775:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760748.787796:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495835264:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.820277:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973209 gen 2 00000008:00010000:3.0:1789760748.820629:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760748.820712:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.820725:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.820751:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760748.820785:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495835264:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.820919:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5892, 5892] 00000080:00200000:2.0:1789760748.821000:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.821007:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.821025:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24137728 i_blocks 49159 00000020:00200000:2.0:1789760748.821029:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.821221:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.821268:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24137728, count: 4096 00000080:00200000:2.0:1789760748.821276:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24137728, bytes: 4096 00000080:00200000:2.0:1789760748.821288:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.821297:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.821299:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24137728, 24141824) 0 0 00000080:00200000:2.0:1789760748.821304:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.821322:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5893, 5893] 00020000:00200000:2.0:1789760748.821330:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24137728 chunk: [24137728, 24141824) 24141824, 4096 00020000:00200000:2.0:1789760748.821335:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.821357:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5893, 5893] 00000008:00010000:2.0:1789760748.821376:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.821382:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.821394:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24137728, 24141824) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.821409:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24137728=0x1705000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.821471:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.821474:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24141824 00000020:00010000:2.0:1789760748.821482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.821486:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.821489:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.821492:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.821494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.821502:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.821509:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.821527:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760748.821687:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 24137728<>24141824 00000100:00100000:3.0:1789760748.821756:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760748.821774:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495835392:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.851734:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973210 gen 2 00000008:00010000:3.0:1789760748.851787:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760748.851797:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.851809:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.851942:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760748.851973:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495835392:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.852216:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5893, 5893] 00000080:00200000:2.0:1789760748.852234:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.852240:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.852255:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24141824 i_blocks 49159 00000020:00200000:2.0:1789760748.852259:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.852268:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.852302:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24141824, count: 4096 00000080:00200000:2.0:1789760748.852310:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24141824, bytes: 4096 00000080:00200000:2.0:1789760748.852321:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.852330:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.852333:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24141824, 24145920) 0 0 00000080:00200000:2.0:1789760748.852338:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.852356:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5894, 5894] 00020000:00200000:2.0:1789760748.852365:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24141824 chunk: [24141824, 24145920) 24145920, 4096 00020000:00200000:2.0:1789760748.852371:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.852393:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5894, 5894] 00000008:00010000:2.0:1789760748.852413:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.852419:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.852431:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24141824, 24145920) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.852456:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24141824=0x1706000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.852514:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.852517:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24145920 00000020:00010000:2.0:1789760748.852525:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.852529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.852532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.852535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.852538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.852546:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.852564:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.852585:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760748.852709:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 24141824<>24145920 00000100:00100000:3.0:1789760748.852773:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495835520:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.901568:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973211 gen 2 00000008:00010000:3.0:1789760748.901621:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760748.901630:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.901642:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.901679:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760748.901713:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495835520:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760748.902466:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5894, 5894] 00000080:00200000:0.0:1789760748.902485:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760748.902492:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760748.902508:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24145920 i_blocks 49159 00000020:00200000:0.0:1789760748.902511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760748.902522:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760748.902577:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24145920, count: 4096 00000080:00200000:0.0:1789760748.902585:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24145920, bytes: 4096 00000080:00200000:0.0:1789760748.902602:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760748.902611:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760748.902614:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24145920, 24150016) 0 0 00000080:00200000:0.0:1789760748.902619:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760748.902637:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5895, 5895] 00020000:00200000:0.0:1789760748.902645:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24145920 chunk: [24145920, 24150016) 24150016, 4096 00020000:00200000:0.0:1789760748.902650:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760748.902672:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5895, 5895] 00000008:00010000:0.0:1789760748.902693:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760748.902699:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760748.902711:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24145920, 24150016) append 0 append lockpos 0 00000080:00200000:0.0:1789760748.902725:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24145920=0x1707000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760748.902789:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760748.902792:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24150016 00000020:00010000:0.0:1789760748.902801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760748.902805:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.902808:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760748.902811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760748.902814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760748.902823:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760748.902830:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760748.902848:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760748.903117:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 24145920<>24150016 00000100:00100000:1.0:1789760748.903204:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495835648:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760748.949864:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760748.949939:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760748.949952:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760748.951422:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760748.951471:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495835648:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760748.951964:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5895, 5895] 00000080:00200000:3.0:1789760748.951982:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760748.951989:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760748.952018:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24150016 i_blocks 49159 00000020:00200000:3.0:1789760748.952023:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760748.952034:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760748.952068:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24150016, count: 4096 00000080:00200000:3.0:1789760748.952075:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24150016, bytes: 4096 00000080:00200000:3.0:1789760748.952089:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760748.952099:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760748.952101:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24150016, 24154112) 0 0 00000080:00200000:3.0:1789760748.952106:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760748.952123:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5896, 5896] 00020000:00200000:3.0:1789760748.952131:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24150016 chunk: [24150016, 24154112) 24154112, 4096 00020000:00200000:3.0:1789760748.952136:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760748.952162:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5896, 5896] 00000008:00010000:3.0:1789760748.952197:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760748.952204:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760748.952217:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24150016, 24154112) append 0 append lockpos 0 00000080:00200000:3.0:1789760748.952231:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24150016=0x1708000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760748.952291:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760748.952294:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24154112 00000020:00010000:3.0:1789760748.952303:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760748.952306:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.952309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.952312:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760748.952315:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760748.952323:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760748.952329:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760748.952350:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760748.952485:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 24150016<>24154112 00000100:00100000:3.0:1789760748.952658:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760748.952747:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495835776:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760748.992153:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973213 gen 2 00000008:00010000:3.0:1789760748.992244:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760748.992252:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.992263:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.992283:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760748.992312:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495835776:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760748.992322:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5896, 5896] 00000080:00200000:2.0:1789760748.992339:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.992345:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.992360:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24154112 i_blocks 49159 00000020:00200000:2.0:1789760748.992364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.992373:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.992404:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24154112, count: 4096 00000080:00200000:2.0:1789760748.992411:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24154112, bytes: 4096 00000080:00200000:2.0:1789760748.992421:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.992430:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.992432:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24154112, 24158208) 0 0 00000080:00200000:2.0:1789760748.992436:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.992453:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5897, 5897] 00020000:00200000:2.0:1789760748.992461:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24154112 chunk: [24154112, 24158208) 24158208, 4096 00020000:00200000:2.0:1789760748.992466:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.992487:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5897, 5897] 00000008:00010000:2.0:1789760748.992505:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.992510:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760748.992520:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24154112, 24158208) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.992535:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24154112=0x1709000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.992603:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.992606:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24158208 00000020:00010000:2.0:1789760748.992615:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.992619:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.992622:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.992626:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.992628:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.992635:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.992642:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.992658:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760748.993147:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 24154112<>24158208 00000100:00100000:2.0:1789760748.993237:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760748.993255:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495835904:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760749.004621:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760749.033947:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973214 gen 2 00000008:00010000:2.0:1789760749.033994:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760749.034001:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760749.034012:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760749.034040:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760749.034067:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495835904:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760749.034485:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5897, 5897] 00000080:00200000:0.0:1789760749.034501:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760749.034507:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760749.034523:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24158208 i_blocks 49159 00000020:00200000:0.0:1789760749.034526:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760749.034610:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760749.034644:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24158208, count: 4096 00000080:00200000:0.0:1789760749.034652:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24158208, bytes: 4096 00000080:00200000:0.0:1789760749.034662:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760749.034670:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760749.034673:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24158208, 24162304) 0 0 00000080:00200000:0.0:1789760749.034677:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760749.034694:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5898, 5898] 00020000:00200000:0.0:1789760749.034702:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24158208 chunk: [24158208, 24162304) 24162304, 4096 00020000:00200000:0.0:1789760749.034707:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760749.034728:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5898, 5898] 00000008:00010000:0.0:1789760749.034758:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760749.034764:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760749.034775:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24158208, 24162304) append 0 append lockpos 0 00000080:00200000:0.0:1789760749.034792:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24158208=0x170a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760749.035471:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760749.035478:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24162304 00000020:00010000:0.0:1789760749.035490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760749.035494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.035498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.035502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760749.035505:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760749.035514:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760749.035522:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760749.035631:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760749.035824:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 24158208<>24162304 00000100:00100000:1.0:1789760749.036004:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495836032:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760749.072227:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973215 gen 2 00000008:00010000:0.0:1789760749.072334:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760749.072342:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760749.072354:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760749.072377:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760749.072404:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495836032:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.072524:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5898, 5898] 00000080:00200000:2.0:1789760749.072541:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.072548:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.072574:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24162304 i_blocks 49159 00000020:00200000:2.0:1789760749.072578:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.072589:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.072624:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24162304, count: 4096 00000080:00200000:2.0:1789760749.072632:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24162304, bytes: 4096 00000080:00200000:2.0:1789760749.072643:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.072652:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.072654:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24162304, 24166400) 0 0 00000080:00200000:2.0:1789760749.072659:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.072677:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5899, 5899] 00020000:00200000:2.0:1789760749.072685:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24162304 chunk: [24162304, 24166400) 24166400, 4096 00020000:00200000:2.0:1789760749.072690:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.072710:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5899, 5899] 00000008:00010000:2.0:1789760749.072728:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.072734:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.072746:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24162304, 24166400) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.072760:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24162304=0x170b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.072814:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.072817:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24166400 00000020:00010000:2.0:1789760749.072826:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.072830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.072833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.072837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.072840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.072847:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.072854:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.072870:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760749.073009:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 24162304<>24166400 00000100:00100000:3.0:1789760749.073254:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495836160:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760749.116988:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760749.117005:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.117018:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.117043:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760749.117073:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495836160:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.117362:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5899, 5899] 00000080:00200000:2.0:1789760749.117379:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.117385:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.117400:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24166400 i_blocks 49159 00000020:00200000:2.0:1789760749.117405:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.117415:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.117450:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24166400, count: 4096 00000080:00200000:2.0:1789760749.117457:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24166400, bytes: 4096 00000080:00200000:2.0:1789760749.117466:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.117477:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.117479:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24166400, 24170496) 0 0 00000080:00200000:2.0:1789760749.117483:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.117501:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5900, 5900] 00020000:00200000:2.0:1789760749.117510:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24166400 chunk: [24166400, 24170496) 24170496, 4096 00020000:00200000:2.0:1789760749.117515:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.117623:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5900, 5900] 00000008:00010000:2.0:1789760749.117641:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.117647:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.117658:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24166400, 24170496) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.117672:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24166400=0x170c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.117731:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.117734:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24170496 00000020:00010000:2.0:1789760749.117743:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.117746:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.117749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.117753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.117756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.117763:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.117771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.117786:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760749.118015:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 24166400<>24170496 00000100:00100000:3.0:1789760749.118230:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760749.118250:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495836288:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.162061:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973217 gen 2 00000008:00010000:3.0:1789760749.162223:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760749.162232:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.162242:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.162265:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760749.162295:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495836288:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.162502:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5900, 5900] 00000080:00200000:2.0:1789760749.162520:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.162611:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.162630:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24170496 i_blocks 49159 00000020:00200000:2.0:1789760749.162634:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.162645:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.162679:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24170496, count: 4096 00000080:00200000:2.0:1789760749.162687:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24170496, bytes: 4096 00000080:00200000:2.0:1789760749.162697:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.162706:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.162709:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24170496, 24174592) 0 0 00000080:00200000:2.0:1789760749.162714:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.162732:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5901, 5901] 00020000:00200000:2.0:1789760749.162741:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24170496 chunk: [24170496, 24174592) 24174592, 4096 00020000:00200000:2.0:1789760749.162745:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.162766:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5901, 5901] 00000008:00010000:2.0:1789760749.162785:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.162791:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.162803:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24170496, 24174592) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.162819:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24170496=0x170d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.162890:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.162893:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24174592 00000020:00010000:2.0:1789760749.162902:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.162905:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.162909:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.162913:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.162916:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.162923:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.162931:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.162946:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760749.163068:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 24170496<>24174592 00000100:00100000:3.0:1789760749.163610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495836416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.198845:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973218 gen 2 00000008:00010000:3.0:1789760749.198899:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760749.198907:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.198917:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.198939:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760749.198966:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495836416:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.199603:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5901, 5901] 00000080:00200000:2.0:1789760749.199621:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.199628:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.199645:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24174592 i_blocks 49159 00000020:00200000:2.0:1789760749.199648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.199658:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.199692:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24174592, count: 4096 00000080:00200000:2.0:1789760749.199699:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24174592, bytes: 4096 00000080:00200000:2.0:1789760749.199710:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.199720:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.199722:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24174592, 24178688) 0 0 00000080:00200000:2.0:1789760749.199727:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.199742:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5902, 5902] 00020000:00200000:2.0:1789760749.199750:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24174592 chunk: [24174592, 24178688) 24178688, 4096 00020000:00200000:2.0:1789760749.199755:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.199775:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5902, 5902] 00000008:00010000:2.0:1789760749.199793:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.199799:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.199810:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24174592, 24178688) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.199822:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24174592=0x170e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.199897:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.199900:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24178688 00000020:00010000:2.0:1789760749.199908:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.199911:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.199914:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.199917:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.199920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.199928:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.199935:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.199953:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760749.200077:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 24174592<>24178688 00000100:00100000:3.0:1789760749.200323:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495836544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.242264:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973219 gen 2 00000008:00010000:3.0:1789760749.242309:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760749.242522:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.242532:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.242703:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760749.242737:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495836544:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760749.242982:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5902, 5902] 00000080:00200000:0.0:1789760749.243000:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760749.243006:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760749.243021:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24178688 i_blocks 49159 00000020:00200000:0.0:1789760749.243025:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760749.243034:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760749.243066:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24178688, count: 4096 00000080:00200000:0.0:1789760749.243074:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24178688, bytes: 4096 00000080:00200000:0.0:1789760749.243085:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760749.243093:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760749.243095:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24178688, 24182784) 0 0 00000080:00200000:0.0:1789760749.243100:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760749.243275:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5903, 5903] 00020000:00200000:0.0:1789760749.243283:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24178688 chunk: [24178688, 24182784) 24182784, 4096 00020000:00200000:0.0:1789760749.243289:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760749.243309:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5903, 5903] 00000008:00010000:0.0:1789760749.243327:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760749.243333:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760749.243345:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24178688, 24182784) append 0 append lockpos 0 00000080:00200000:0.0:1789760749.243360:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24178688=0x170f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760749.243420:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760749.243423:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24182784 00000020:00010000:0.0:1789760749.243431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760749.243435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.243438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.243441:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760749.243444:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760749.243451:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760749.243458:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760749.243473:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760749.243799:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 24178688<>24182784 00000100:00100000:1.0:1789760749.243888:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495836672:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760749.294209:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973220 gen 2 00000008:00010000:0.0:1789760749.294260:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760749.294269:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760749.294281:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760749.294358:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760749.294387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495836672:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.294622:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5903, 5903] 00000080:00200000:2.0:1789760749.294638:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.294645:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.294662:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24182784 i_blocks 49159 00000020:00200000:2.0:1789760749.294666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.294676:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.294712:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24182784, count: 4096 00000080:00200000:2.0:1789760749.294719:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24182784, bytes: 4096 00000080:00200000:2.0:1789760749.294730:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.294739:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.294741:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24182784, 24186880) 0 0 00000080:00200000:2.0:1789760749.294746:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.294764:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5904, 5904] 00020000:00200000:2.0:1789760749.294772:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24182784 chunk: [24182784, 24186880) 24186880, 4096 00020000:00200000:2.0:1789760749.294777:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.294796:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5904, 5904] 00000008:00010000:2.0:1789760749.294957:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.294964:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.294977:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24182784, 24186880) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.294993:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24182784=0x1710000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.297751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.297763:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24186880 00000020:00010000:2.0:1789760749.297784:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.297789:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.297794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.297798:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.297801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.297824:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.297835:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.297878:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760749.298025:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 24182784<>24186880 00000100:00100000:3.0:1789760749.298345:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495836800:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.334921:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973221 gen 2 00000008:00010000:3.0:1789760749.334972:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760749.334980:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.334991:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.335016:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760749.335046:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495836800:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.336511:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5904, 5904] 00000080:00200000:2.0:1789760749.336668:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.336676:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.336693:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24186880 i_blocks 49159 00000020:00200000:2.0:1789760749.336698:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.336711:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.336746:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24186880, count: 4096 00000080:00200000:2.0:1789760749.336755:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24186880, bytes: 4096 00000080:00200000:2.0:1789760749.336768:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.336778:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.336781:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24186880, 24190976) 0 0 00000080:00200000:2.0:1789760749.336785:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.336802:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5905, 5905] 00020000:00200000:2.0:1789760749.336811:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24186880 chunk: [24186880, 24190976) 24190976, 4096 00020000:00200000:2.0:1789760749.336817:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.336837:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5905, 5905] 00000008:00010000:2.0:1789760749.336983:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.336990:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.337003:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24186880, 24190976) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.337017:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24186880=0x1711000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.337083:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.337086:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24190976 00000020:00010000:2.0:1789760749.337094:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.337097:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.337100:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.337104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.337107:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.337115:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.337121:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.337142:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760749.337402:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 24186880<>24190976 00000100:00100000:3.0:1789760749.337482:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495836928:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.369431:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973222 gen 2 00000008:00010000:3.0:1789760749.369480:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760749.369488:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.369499:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.369522:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760749.369568:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495836928:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760749.369745:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5905, 5905] 00000080:00200000:1.0:1789760749.369763:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760749.369769:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760749.369784:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24190976 i_blocks 49159 00000020:00200000:1.0:1789760749.369788:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760749.369797:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760749.369830:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24190976, count: 4096 00000080:00200000:1.0:1789760749.369837:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24190976, bytes: 4096 00000080:00200000:1.0:1789760749.369846:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760749.369962:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760749.369965:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24190976, 24195072) 0 0 00000080:00200000:1.0:1789760749.369970:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760749.369987:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5906, 5906] 00020000:00200000:1.0:1789760749.369995:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24190976 chunk: [24190976, 24195072) 24195072, 4096 00020000:00200000:1.0:1789760749.370000:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760749.370020:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5906, 5906] 00000008:00010000:1.0:1789760749.370039:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760749.370045:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760749.370057:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24190976, 24195072) append 0 append lockpos 0 00000080:00200000:1.0:1789760749.370071:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24190976=0x1712000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760749.370127:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760749.370131:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24195072 00000020:00010000:1.0:1789760749.370138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:1.0:1789760749.370141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.370144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.370148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760749.370151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760749.370157:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760749.370163:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760749.370493:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760749.370691:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24190976<>24195072 00000100:00100000:1.0:1789760749.370736:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760749.370754:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495837056:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760749.409534:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973223 gen 2 00000008:00010000:0.0:1789760749.409601:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760749.409610:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760749.409621:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760749.409637:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760749.409663:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495837056:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760749.409801:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5906, 5906] 00000080:00200000:3.0:1789760749.409817:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760749.409823:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760749.409839:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24195072 i_blocks 49159 00000020:00200000:3.0:1789760749.409843:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760749.409852:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760749.409882:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24195072, count: 4096 00000080:00200000:3.0:1789760749.409888:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24195072, bytes: 4096 00000080:00200000:3.0:1789760749.409897:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760749.409905:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760749.409908:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24195072, 24199168) 0 0 00000080:00200000:3.0:1789760749.409912:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760749.409928:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5907, 5907] 00020000:00200000:3.0:1789760749.409937:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24195072 chunk: [24195072, 24199168) 24199168, 4096 00020000:00200000:3.0:1789760749.409942:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760749.409960:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5907, 5907] 00000008:00010000:3.0:1789760749.409978:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760749.409983:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760749.409995:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24195072, 24199168) append 0 append lockpos 0 00000080:00200000:3.0:1789760749.410008:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24195072=0x1713000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760749.410062:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760749.410065:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24199168 00000020:00010000:3.0:1789760749.410074:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:3.0:1789760749.410082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.410086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.410089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760749.410092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760749.410100:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760749.410106:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760749.410124:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760749.410253:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 24195072<>24199168 00000100:00100000:3.0:1789760749.410294:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760749.410309:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495837184:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.438603:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973224 gen 2 00000008:00010000:3.0:1789760749.438653:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760749.438661:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.438672:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.438758:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760749.438788:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495837184:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.438872:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5907, 5907] 00000080:00200000:2.0:1789760749.439015:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.439023:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.439037:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24199168 i_blocks 49159 00000020:00200000:2.0:1789760749.439040:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.439051:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.439237:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24199168, count: 4096 00000080:00200000:2.0:1789760749.439247:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24199168, bytes: 4096 00000080:00200000:2.0:1789760749.439258:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.439268:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.439270:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24199168, 24203264) 0 0 00000080:00200000:2.0:1789760749.439275:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.439292:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5908, 5908] 00020000:00200000:2.0:1789760749.439300:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24199168 chunk: [24199168, 24203264) 24203264, 4096 00020000:00200000:2.0:1789760749.439305:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.439326:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5908, 5908] 00000008:00010000:2.0:1789760749.439344:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.439350:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.439361:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24199168, 24203264) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.439375:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24199168=0x1714000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.439433:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.439437:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24203264 00000020:00010000:2.0:1789760749.439446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.439450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.439454:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.439458:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.439461:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.439469:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.439476:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.439495:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760749.439848:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 24199168<>24203264 00000100:00100000:3.0:1789760749.439935:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495837312:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.475777:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973225 gen 2 00000008:00010000:3.0:1789760749.475827:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760749.475836:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.475848:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760749.475971:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5908, 5908] 00000080:00200000:0.0:1789760749.475991:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760749.475998:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760749.476015:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24203264 i_blocks 49159 00000020:00200000:0.0:1789760749.476019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760749.476030:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760749.476068:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24203264, count: 4096 00000080:00200000:0.0:1789760749.476076:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24203264, bytes: 4096 00000080:00200000:0.0:1789760749.476087:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760749.476097:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760749.476100:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24203264, 24207360) 0 0 00000080:00200000:0.0:1789760749.476105:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760749.476121:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5909, 5909] 00020000:00200000:0.0:1789760749.476128:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24203264 chunk: [24203264, 24207360) 24207360, 4096 00020000:00200000:0.0:1789760749.476133:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760749.476152:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5909, 5909] 00000008:00010000:0.0:1789760749.476203:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760749.476209:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760749.476221:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24203264, 24207360) append 0 append lockpos 0 00000080:00200000:0.0:1789760749.476235:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24203264=0x1715000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760749.476296:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760749.476299:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24207360 00000020:00010000:0.0:1789760749.476307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760749.476311:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.476314:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.476318:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760749.476322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760749.476329:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760749.476336:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760749.476356:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760749.476482:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 24203264<>24207360 00000100:00100000:1.0:1789760749.476576:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495837440:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760749.476734:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760749.476751:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495837312:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760749.514087:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973226 gen 2 00000008:00010000:0.0:1789760749.514140:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760749.514149:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760749.514160:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760749.514276:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760749.514304:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495837440:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.515282:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5909, 5909] 00000080:00200000:2.0:1789760749.515303:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.515310:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.515413:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24207360 i_blocks 49159 00000020:00200000:2.0:1789760749.515418:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.515429:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.515468:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24207360, count: 4096 00000080:00200000:2.0:1789760749.515475:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24207360, bytes: 4096 00000080:00200000:2.0:1789760749.515487:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.515496:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.515498:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24207360, 24211456) 0 0 00000080:00200000:2.0:1789760749.515581:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.515599:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5910, 5910] 00020000:00200000:2.0:1789760749.515607:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24207360 chunk: [24207360, 24211456) 24211456, 4096 00020000:00200000:2.0:1789760749.515719:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.515745:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5910, 5910] 00000008:00010000:2.0:1789760749.515766:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.515773:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.515786:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24207360, 24211456) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.515802:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24207360=0x1716000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.515863:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.515866:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24211456 00000020:00010000:2.0:1789760749.515876:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.515880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.515883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.515886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.515889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.515897:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.515905:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.515921:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760749.516062:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 24207360<>24211456 00000100:00100000:3.0:1789760749.516622:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495837568:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.589080:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973227 gen 2 00000008:00010000:3.0:1789760749.589138:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760749.589147:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.589159:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.589346:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760749.589378:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495837568:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.589715:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5910, 5910] 00000080:00200000:2.0:1789760749.589734:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.589740:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.589757:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24211456 i_blocks 49159 00000020:00200000:2.0:1789760749.589761:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.589771:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.589963:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24211456, count: 4096 00000080:00200000:2.0:1789760749.589971:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24211456, bytes: 4096 00000080:00200000:2.0:1789760749.589986:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.589998:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.590000:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24211456, 24215552) 0 0 00000080:00200000:2.0:1789760749.590005:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.590025:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5911, 5911] 00020000:00200000:2.0:1789760749.590033:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24211456 chunk: [24211456, 24215552) 24215552, 4096 00020000:00200000:2.0:1789760749.590037:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.590057:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5911, 5911] 00000008:00010000:2.0:1789760749.590076:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.590082:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.590093:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24211456, 24215552) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.590107:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24211456=0x1717000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.590192:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.590196:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24215552 00000020:00010000:2.0:1789760749.590204:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.590207:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.590211:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.590214:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.590217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.590225:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.590233:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.590252:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760749.590371:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 24211456<>24215552 00000100:00100000:3.0:1789760749.590430:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760749.590451:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495837696:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760749.638974:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760749.638989:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760749.639001:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760749.639031:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760749.639060:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495837696:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760749.639195:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5911, 5911] 00000080:00200000:0.0:1789760749.639211:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760749.639218:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760749.639232:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24215552 i_blocks 49159 00000020:00200000:0.0:1789760749.639235:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760749.639244:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760749.639279:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24215552, count: 4096 00000080:00200000:0.0:1789760749.639286:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24215552, bytes: 4096 00000080:00200000:0.0:1789760749.639296:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760749.639304:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760749.639307:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24215552, 24219648) 0 0 00000080:00200000:0.0:1789760749.639311:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760749.639326:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5912, 5912] 00020000:00200000:0.0:1789760749.639334:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24215552 chunk: [24215552, 24219648) 24219648, 4096 00020000:00200000:0.0:1789760749.639338:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760749.639356:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5912, 5912] 00000008:00010000:0.0:1789760749.639373:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760749.639378:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760749.639389:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24215552, 24219648) append 0 append lockpos 0 00000080:00200000:0.0:1789760749.639403:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24215552=0x1718000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760749.639456:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760749.639459:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24219648 00000020:00010000:0.0:1789760749.639467:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760749.639470:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.639473:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.639476:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760749.639478:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760749.639485:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760749.639491:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760749.639506:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760749.640000:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 24215552<>24219648 00000100:00100000:1.0:1789760749.640084:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495837824:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760749.705349:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973228 gen 2 00000100:00100000:0.0:1789760749.705391:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211836a00 x1876699495837824/t4294973229(4294973229) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760765 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760749.705442:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760749.705449:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760749.705460:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760749.705488:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760749.705512:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495837824:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.705742:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5912, 5912] 00000080:00200000:2.0:1789760749.705760:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.705767:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.705783:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24219648 i_blocks 49159 00000020:00200000:2.0:1789760749.705786:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.705797:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.705831:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24219648, count: 4096 00000080:00200000:2.0:1789760749.705839:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24219648, bytes: 4096 00000080:00200000:2.0:1789760749.705863:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.705970:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.705973:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24219648, 24223744) 0 0 00000080:00200000:2.0:1789760749.705978:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.705996:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5913, 5913] 00020000:00200000:2.0:1789760749.706005:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24219648 chunk: [24219648, 24223744) 24223744, 4096 00020000:00200000:2.0:1789760749.706010:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.706033:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5913, 5913] 00000008:00010000:2.0:1789760749.706051:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.706057:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.706069:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24219648, 24223744) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.706085:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24219648=0x1719000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.706144:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.706147:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24223744 00000020:00010000:2.0:1789760749.706156:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.706160:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.706163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.706166:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.706220:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.706229:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.706239:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.706260:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760749.706395:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 24219648<>24223744 00000100:00100000:3.0:1789760749.706477:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495837952:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760749.745318:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973230 gen 2 00000008:00010000:2.0:1789760749.745395:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760749.745404:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760749.745417:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760749.745447:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760749.745483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495837952:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760749.745674:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5913, 5913] 00000080:00200000:0.0:1789760749.745691:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760749.745697:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760749.745714:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24223744 i_blocks 49159 00000020:00200000:0.0:1789760749.745718:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760749.745728:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760749.745765:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24223744, count: 4096 00000080:00200000:0.0:1789760749.745773:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24223744, bytes: 4096 00000080:00200000:0.0:1789760749.745783:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760749.745792:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760749.745794:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24223744, 24227840) 0 0 00000080:00200000:0.0:1789760749.745799:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760749.745816:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5914, 5914] 00020000:00200000:0.0:1789760749.745835:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24223744 chunk: [24223744, 24227840) 24227840, 4096 00020000:00200000:0.0:1789760749.745840:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760749.746018:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5914, 5914] 00000008:00010000:0.0:1789760749.746037:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760749.746043:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760749.746056:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24223744, 24227840) append 0 append lockpos 0 00000080:00200000:0.0:1789760749.746072:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24223744=0x171a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760749.746132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760749.746135:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24227840 00000020:00010000:0.0:1789760749.746143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:0.0:1789760749.746147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.746150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760749.746154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760749.746156:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760749.746162:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760749.746281:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760749.746300:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760749.746432:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 24223744<>24227840 00000100:00100000:1.0:1789760749.746505:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495838080:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760749.785576:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973231 gen 2 00000008:00010000:0.0:1789760749.785631:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760749.785641:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760749.785768:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760749.785794:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760749.785824:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495838080:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.785990:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5914, 5914] 00000080:00200000:2.0:1789760749.786006:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.786013:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.786029:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24227840 i_blocks 49159 00000020:00200000:2.0:1789760749.786033:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.786043:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.786077:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24227840, count: 4096 00000080:00200000:2.0:1789760749.786084:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24227840, bytes: 4096 00000080:00200000:2.0:1789760749.786094:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.786103:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.786105:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24227840, 24231936) 0 0 00000080:00200000:2.0:1789760749.786109:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.786127:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5915, 5915] 00020000:00200000:2.0:1789760749.786135:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24227840 chunk: [24227840, 24231936) 24231936, 4096 00020000:00200000:2.0:1789760749.786140:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.786162:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5915, 5915] 00000008:00010000:2.0:1789760749.786205:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.786211:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.786224:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24227840, 24231936) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.786238:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24227840=0x171b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.786297:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.786300:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24231936 00000020:00010000:2.0:1789760749.786309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.786313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.786316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.786319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.786322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.786330:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.786338:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.786354:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760749.786487:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 24227840<>24231936 00000100:00100000:3.0:1789760749.786573:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495838208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.843425:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973232 gen 2 00000008:00010000:3.0:1789760749.843484:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760749.843492:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.843504:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.843539:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760749.843577:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495838208:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.843819:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5915, 5915] 00000080:00200000:2.0:1789760749.843843:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.843857:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.843978:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24231936 i_blocks 49159 00000020:00200000:2.0:1789760749.843983:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.844000:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.844042:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24231936, count: 4096 00000080:00200000:2.0:1789760749.844049:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24231936, bytes: 4096 00000080:00200000:2.0:1789760749.844059:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.844066:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.844069:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24231936, 24236032) 0 0 00000080:00200000:2.0:1789760749.844073:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.844089:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5916, 5916] 00020000:00200000:2.0:1789760749.844097:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24231936 chunk: [24231936, 24236032) 24236032, 4096 00020000:00200000:2.0:1789760749.844102:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.844122:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5916, 5916] 00000008:00010000:2.0:1789760749.844141:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.844146:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.844158:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24231936, 24236032) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.844201:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24231936=0x171c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.844279:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.844283:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24236032 00000020:00010000:2.0:1789760749.844291:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.844294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.844301:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.844304:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.844307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.844316:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.844324:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.844340:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760749.844477:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 24231936<>24236032 00000100:00100000:3.0:1789760749.845014:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495838336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.880316:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973233 gen 2 00000008:00010000:3.0:1789760749.880365:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760749.880374:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.880391:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.880415:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760749.880447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495838336:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.880913:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5916, 5916] 00000080:00200000:2.0:1789760749.880932:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.880938:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.880954:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24236032 i_blocks 49159 00000020:00200000:2.0:1789760749.880957:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.880967:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.881002:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24236032, count: 4096 00000080:00200000:2.0:1789760749.881011:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24236032, bytes: 4096 00000080:00200000:2.0:1789760749.881021:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.881029:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.881032:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24236032, 24240128) 0 0 00000080:00200000:2.0:1789760749.881036:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.881052:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5917, 5917] 00020000:00200000:2.0:1789760749.881061:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24236032 chunk: [24236032, 24240128) 24240128, 4096 00020000:00200000:2.0:1789760749.881065:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.881089:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5917, 5917] 00000008:00010000:2.0:1789760749.881108:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.881114:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.881125:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24236032, 24240128) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.881140:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24236032=0x171d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.881239:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.881242:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24240128 00000020:00010000:2.0:1789760749.881251:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.881255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.881258:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.881262:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.881265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.881273:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.881280:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.881297:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813480 00000008:00100000:2.0:1789760749.881436:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 24236032<>24240128 00000100:00100000:3.0:1789760749.881758:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495838464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.911515:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973234 gen 2 00000008:00010000:3.0:1789760749.911577:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813480 00000020:00200000:3.0:1789760749.911585:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.911596:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.911684:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813480 00000100:00100000:3.0:1789760749.911715:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495838464:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.911978:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5917, 5917] 00000080:00200000:2.0:1789760749.911994:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.912000:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.912017:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24240128 i_blocks 49159 00000020:00200000:2.0:1789760749.912020:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.912030:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.912066:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24240128, count: 4096 00000080:00200000:2.0:1789760749.912074:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24240128, bytes: 4096 00000080:00200000:2.0:1789760749.912083:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.912091:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.912094:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24240128, 24244224) 0 0 00000080:00200000:2.0:1789760749.912098:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.912116:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5918, 5918] 00020000:00200000:2.0:1789760749.912124:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24240128 chunk: [24240128, 24244224) 24244224, 4096 00020000:00200000:2.0:1789760749.912129:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.912150:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5918, 5918] 00000008:00010000:2.0:1789760749.912205:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.912211:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.912224:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24240128, 24244224) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.912239:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24240128=0x171e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.912569:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.912572:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24244224 00000020:00010000:2.0:1789760749.912582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.912585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.912710:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.912714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.912716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.912727:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.912734:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.912754:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760749.913140:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 24240128<>24244224 00000100:00100000:3.0:1789760749.913236:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495838592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760749.988942:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973235 gen 2 00000008:00010000:3.0:1789760749.988994:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760749.989003:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.989016:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.989075:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760749.989340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495838592:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760749.989489:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5918, 5918] 00000080:00200000:2.0:1789760749.989505:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.989511:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.989599:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24244224 i_blocks 49159 00000020:00200000:2.0:1789760749.989604:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.989614:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.989651:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24244224, count: 4096 00000080:00200000:2.0:1789760749.989657:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24244224, bytes: 4096 00000080:00200000:2.0:1789760749.989668:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.989676:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.989678:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24244224, 24248320) 0 0 00000080:00200000:2.0:1789760749.989682:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.989699:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5919, 5919] 00020000:00200000:2.0:1789760749.989707:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24244224 chunk: [24244224, 24248320) 24248320, 4096 00020000:00200000:2.0:1789760749.989712:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.989733:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5919, 5919] 00000008:00010000:2.0:1789760749.989751:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.989756:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760749.989768:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24244224, 24248320) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.989781:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24244224=0x171f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.989838:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.989841:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24248320 00000020:00010000:2.0:1789760749.989849:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.989853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.989856:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.989860:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.989863:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.989942:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.989948:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.989965:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760749.990098:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 24244224<>24248320 00000100:00100000:3.0:1789760749.990187:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495838720:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760750.028700:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760750.041150:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973236 gen 2 00000008:00010000:3.0:1789760750.041244:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760750.041252:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.041264:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.041286:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760750.041315:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495838720:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.041574:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5919, 5919] 00000080:00200000:1.0:1789760750.041591:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.041597:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.041613:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24248320 i_blocks 49159 00000020:00200000:1.0:1789760750.041617:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.041626:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.041660:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24248320, count: 4096 00000080:00200000:1.0:1789760750.041667:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24248320, bytes: 4096 00000080:00200000:1.0:1789760750.041677:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.041685:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.041687:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24248320, 24252416) 0 0 00000080:00200000:1.0:1789760750.041692:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.041709:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5920, 5920] 00020000:00200000:1.0:1789760750.041718:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24248320 chunk: [24248320, 24252416) 24252416, 4096 00020000:00200000:1.0:1789760750.041722:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.041741:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5920, 5920] 00000008:00010000:1.0:1789760750.041760:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.041766:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.041777:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24248320, 24252416) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.041793:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24248320=0x1720000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.042124:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.042128:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24252416 00000020:00010000:1.0:1789760750.042137:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.042141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.042144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.042147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.042150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.042158:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.042164:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.042410:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760750.042552:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 24248320<>24252416 00000100:00100000:0.0:1789760750.042617:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495838848:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.088317:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973237 gen 2 00000008:00010000:0.0:1789760750.088371:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760750.088380:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.088391:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.088420:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000080:00200000:1.0:1789760750.088611:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5920, 5920] 00000080:00200000:1.0:1789760750.088630:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.088636:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:0.0:1789760750.088646:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495838848:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.088652:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24252416 i_blocks 49159 00000020:00200000:1.0:1789760750.088656:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.088665:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.088699:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24252416, count: 4096 00000080:00200000:1.0:1789760750.088706:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24252416, bytes: 4096 00000080:00200000:1.0:1789760750.088722:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.088731:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.088734:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24252416, 24256512) 0 0 00000080:00200000:1.0:1789760750.088738:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.088756:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5921, 5921] 00020000:00200000:1.0:1789760750.088764:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24252416 chunk: [24252416, 24256512) 24256512, 4096 00020000:00200000:1.0:1789760750.088768:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.088785:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5921, 5921] 00000008:00010000:1.0:1789760750.088801:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.088806:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.088816:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24252416, 24256512) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.088828:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24252416=0x1721000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.088880:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.088882:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24256512 00000020:00010000:1.0:1789760750.088891:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.088894:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.088897:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.088901:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.088904:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.088911:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.088918:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.088936:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760750.089067:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 24252416<>24256512 00000100:00100000:0.0:1789760750.089378:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495838976:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.138928:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973238 gen 2 00000008:00010000:0.0:1789760750.138983:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760750.138993:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.139004:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.139026:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760750.139057:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495838976:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.139163:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5921, 5921] 00000080:00200000:1.0:1789760750.139208:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.139214:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.139230:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24256512 i_blocks 49159 00000020:00200000:1.0:1789760750.139233:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.139242:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.139274:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24256512, count: 4096 00000080:00200000:1.0:1789760750.139281:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24256512, bytes: 4096 00000080:00200000:1.0:1789760750.139291:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.139299:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.139301:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24256512, 24260608) 0 0 00000080:00200000:1.0:1789760750.139305:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.139322:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5922, 5922] 00020000:00200000:1.0:1789760750.139331:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24256512 chunk: [24256512, 24260608) 24260608, 4096 00020000:00200000:1.0:1789760750.139335:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.139354:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5922, 5922] 00000008:00010000:1.0:1789760750.139372:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.139377:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.139388:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24256512, 24260608) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.139402:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24256512=0x1722000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.139459:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.139462:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24260608 00000020:00010000:1.0:1789760750.139472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.139475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.139479:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.139482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.139485:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.139492:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.139499:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.139515:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760750.139657:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 24256512<>24260608 00000100:00100000:0.0:1789760750.140010:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495839104:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.183334:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973239 gen 2 00000008:00010000:0.0:1789760750.183397:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760750.183406:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.183417:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.183447:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760750.183487:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495839104:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760750.183498:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5922, 5922] 00000080:00200000:2.0:1789760750.183517:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.183532:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.183557:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24260608 i_blocks 49159 00000020:00200000:2.0:1789760750.183562:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.183571:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.183607:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24260608, count: 4096 00000080:00200000:2.0:1789760750.183619:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24260608, bytes: 4096 00000080:00200000:2.0:1789760750.183629:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.183637:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.183640:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24260608, 24264704) 0 0 00000080:00200000:2.0:1789760750.183644:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.183662:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5923, 5923] 00020000:00200000:2.0:1789760750.183669:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24260608 chunk: [24260608, 24264704) 24264704, 4096 00020000:00200000:2.0:1789760750.183675:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.183695:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5923, 5923] 00000008:00010000:2.0:1789760750.183714:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.183719:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760750.183732:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24260608, 24264704) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.183747:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24260608=0x1723000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.183808:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.183811:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24264704 00000020:00010000:2.0:1789760750.183819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.183823:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.183826:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.183830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.183834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.183841:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.183853:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.183869:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760750.183997:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 24260608<>24264704 00000100:00100000:3.0:1789760750.184527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495839232:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760750.251630:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760750.251646:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.251659:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.251683:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760750.251712:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495839232:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760750.251809:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5923, 5923] 00000080:00200000:2.0:1789760750.251835:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.251843:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.251861:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24264704 i_blocks 49159 00000020:00200000:2.0:1789760750.251865:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.251875:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.251912:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24264704, count: 4096 00000080:00200000:2.0:1789760750.251920:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24264704, bytes: 4096 00000080:00200000:2.0:1789760750.251931:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.251941:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.251944:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24264704, 24268800) 0 0 00000080:00200000:2.0:1789760750.251948:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.251968:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5924, 5924] 00020000:00200000:2.0:1789760750.251976:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24264704 chunk: [24264704, 24268800) 24268800, 4096 00020000:00200000:2.0:1789760750.251982:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.252004:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5924, 5924] 00000008:00010000:2.0:1789760750.252024:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.252040:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760750.252055:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24264704, 24268800) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.252073:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24264704=0x1724000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.252144:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.252148:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24268800 00000020:00010000:2.0:1789760750.252157:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.252161:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.252164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.252216:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.252221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.252229:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.252237:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.252254:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760750.252388:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 24264704<>24268800 00000100:00100000:2.0:1789760750.252431:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760750.252449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495839360:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760750.289847:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973241 gen 2 00000008:00010000:2.0:1789760750.289909:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760750.289918:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.289928:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.290130:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760750.290162:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495839360:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760750.290357:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5924, 5924] 00000080:00200000:3.0:1789760750.290376:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.290382:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.290476:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24268800 i_blocks 49159 00000020:00200000:2.0:1789760750.290481:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.290491:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.290525:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24268800, count: 4096 00000080:00200000:2.0:1789760750.290533:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24268800, bytes: 4096 00000080:00200000:2.0:1789760750.290554:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.290563:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.290566:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24268800, 24272896) 0 0 00000080:00200000:2.0:1789760750.290570:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.290655:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5925, 5925] 00020000:00200000:2.0:1789760750.290665:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24268800 chunk: [24268800, 24272896) 24272896, 4096 00020000:00200000:2.0:1789760750.290669:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.290690:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5925, 5925] 00000008:00010000:2.0:1789760750.290708:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.290713:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760750.290725:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24268800, 24272896) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.290807:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24268800=0x1725000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.290867:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.290870:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24272896 00000020:00010000:2.0:1789760750.290878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.290881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.290884:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.290887:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0098/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.290890:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.290897:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.290904:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.290949:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760750.291435:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 24268800<>24272896 00000100:00100000:3.0:1789760750.291491:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760750.291702:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495839488:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760750.329901:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973242 gen 2 00000008:00010000:2.0:1789760750.329953:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760750.329962:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.329974:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.329998:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760750.330035:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495839488:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760750.330572:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5925, 5925] 00000080:00200000:3.0:1789760750.330590:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.330596:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.330612:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24272896 i_blocks 49159 00000020:00200000:3.0:1789760750.330615:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.330625:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760750.330690:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24272896, count: 4096 00000080:00200000:3.0:1789760750.330697:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24272896, bytes: 4096 00000080:00200000:3.0:1789760750.330709:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760750.330719:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760750.330721:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24272896, 24276992) 0 0 00000080:00200000:3.0:1789760750.330726:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760750.330744:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5926, 5926] 00020000:00200000:3.0:1789760750.330752:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24272896 chunk: [24272896, 24276992) 24276992, 4096 00020000:00200000:3.0:1789760750.330756:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760750.330777:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5926, 5926] 00000008:00010000:3.0:1789760750.330796:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760750.330801:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760750.330813:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24272896, 24276992) append 0 append lockpos 0 00000080:00200000:3.0:1789760750.330827:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24272896=0x1726000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760750.334972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760750.334986:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24276992 00000020:00010000:3.0:1789760750.335005:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760750.335010:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.335214:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.335218:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760750.335221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760750.335235:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760750.335246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760750.335281:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760750.335428:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 24272896<>24276992 00000100:00100000:2.0:1789760750.335804:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760750.335833:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495839616:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760750.384292:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973243 gen 2 00000008:00010000:2.0:1789760750.384343:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760750.384353:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.384366:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.384389:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760750.384420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495839616:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760750.384746:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5926, 5926] 00000080:00200000:3.0:1789760750.384764:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.384771:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.384790:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24276992 i_blocks 49159 00000020:00200000:3.0:1789760750.384793:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.384804:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760750.384839:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24276992, count: 4096 00000080:00200000:3.0:1789760750.384846:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24276992, bytes: 4096 00000080:00200000:3.0:1789760750.384855:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760750.384864:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760750.384866:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24276992, 24281088) 0 0 00000080:00200000:3.0:1789760750.384871:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760750.384889:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5927, 5927] 00020000:00200000:3.0:1789760750.384897:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24276992 chunk: [24276992, 24281088) 24281088, 4096 00020000:00200000:3.0:1789760750.384901:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760750.384920:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5927, 5927] 00000008:00010000:3.0:1789760750.384937:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760750.384943:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760750.384954:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24276992, 24281088) append 0 append lockpos 0 00000080:00200000:3.0:1789760750.384968:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24276992=0x1727000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760750.385029:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760750.385032:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24281088 00000020:00010000:3.0:1789760750.385039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760750.385043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.385046:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.385049:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760750.385053:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760750.386209:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760750.386224:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760750.386255:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760750.386456:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 24276992<>24281088 00000100:00100000:2.0:1789760750.386778:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495839744:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760750.429255:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973244 gen 2 00000008:00010000:2.0:1789760750.429310:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760750.429318:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.429330:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.429351:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760750.429381:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495839744:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760750.430011:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5927, 5927] 00000080:00200000:3.0:1789760750.430030:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.430037:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.430055:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24281088 i_blocks 49159 00000020:00200000:3.0:1789760750.430059:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.430068:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.432362:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24281088, count: 4096 00000080:00200000:2.0:1789760750.432385:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24281088, bytes: 4096 00000080:00200000:2.0:1789760750.432403:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.432414:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.432417:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24281088, 24285184) 0 0 00000080:00200000:2.0:1789760750.432422:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.432443:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5928, 5928] 00020000:00200000:2.0:1789760750.432453:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24281088 chunk: [24281088, 24285184) 24285184, 4096 00020000:00200000:2.0:1789760750.432458:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.432516:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5928, 5928] 00000008:00010000:2.0:1789760750.432537:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.432553:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760750.432566:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24281088, 24285184) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.432582:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24281088=0x1728000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.432645:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.432649:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24285184 00000020:00010000:2.0:1789760750.432658:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.432662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.432665:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.432668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.432671:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.432682:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.432689:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.432728:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760750.432985:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 24281088<>24285184 00000100:00100000:3.0:1789760750.433072:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495839872:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760750.518758:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973245 gen 2 00000008:00010000:2.0:1789760750.518932:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:2.0:1789760750.518940:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.518951:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.519019:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:2.0:1789760750.519053:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495839872:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.523566:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5928, 5928] 00000080:00200000:1.0:1789760750.523661:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.523669:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.523689:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24285184 i_blocks 49159 00000020:00200000:1.0:1789760750.523695:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.523715:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.523756:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24285184, count: 4096 00000080:00200000:1.0:1789760750.523765:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24285184, bytes: 4096 00000080:00200000:1.0:1789760750.523848:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.523858:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.523917:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24285184, 24289280) 0 0 00000080:00200000:1.0:1789760750.523923:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.523941:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5929, 5929] 00020000:00200000:1.0:1789760750.523950:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24285184 chunk: [24285184, 24289280) 24289280, 4096 00020000:00200000:1.0:1789760750.523955:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.523979:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5929, 5929] 00000008:00010000:1.0:1789760750.523999:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.524004:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.524016:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24285184, 24289280) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.524033:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24285184=0x1729000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.524108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.524111:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24289280 00000020:00010000:1.0:1789760750.524120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.524124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.524127:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.524131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.524135:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.524143:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.524151:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.524221:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760750.524354:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 24285184<>24289280 00000100:00100000:0.0:1789760750.524685:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840000:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.593245:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973246 gen 2 00000008:00010000:0.0:1789760750.593311:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760750.593320:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.593332:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.593355:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760750.593387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840000:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.593545:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5929, 5929] 00000080:00200000:1.0:1789760750.593561:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.593568:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.593583:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24289280 i_blocks 49159 00000020:00200000:1.0:1789760750.593586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.593594:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.593628:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24289280, count: 4096 00000080:00200000:1.0:1789760750.593635:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24289280, bytes: 4096 00000080:00200000:1.0:1789760750.593655:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.593663:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.593666:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24289280, 24293376) 0 0 00000080:00200000:1.0:1789760750.593670:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.593685:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5930, 5930] 00020000:00200000:1.0:1789760750.593693:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24289280 chunk: [24289280, 24293376) 24293376, 4096 00020000:00200000:1.0:1789760750.593698:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.593716:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5930, 5930] 00000008:00010000:1.0:1789760750.593734:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.593740:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.593752:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24289280, 24293376) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.593765:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24289280=0x172a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.593821:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.593824:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24293376 00000020:00010000:1.0:1789760750.593832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.593835:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.593838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.593842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.593845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.593853:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.594013:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.594032:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760750.594158:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24289280<>24293376 00000100:00100000:0.0:1789760750.594379:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495840128:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.670954:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973247 gen 2 00000008:00010000:0.0:1789760750.671005:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760750.671013:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.671026:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.671046:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760750.671086:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495840128:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.671320:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5930, 5930] 00000080:00200000:1.0:1789760750.671337:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.671344:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.671360:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24293376 i_blocks 49159 00000020:00200000:1.0:1789760750.671364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.671375:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.671409:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24293376, count: 4096 00000080:00200000:1.0:1789760750.671417:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24293376, bytes: 4096 00000080:00200000:1.0:1789760750.671428:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.671437:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.671440:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24293376, 24297472) 0 0 00000080:00200000:1.0:1789760750.671444:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.671461:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5931, 5931] 00020000:00200000:1.0:1789760750.671470:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24293376 chunk: [24293376, 24297472) 24297472, 4096 00020000:00200000:1.0:1789760750.671475:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.671496:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5931, 5931] 00000008:00010000:1.0:1789760750.671515:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.671521:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.671533:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24293376, 24297472) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.671558:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24293376=0x172b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.671628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.671630:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24297472 00000020:00010000:1.0:1789760750.671638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.671641:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.671644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.671648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.671650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.671658:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.671675:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.671692:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760750.671818:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 24293376<>24297472 00000100:00100000:0.0:1789760750.672271:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760750.672293:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495840256:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760750.778612:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760750.778629:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.778641:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.778677:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760750.778708:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495840256:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.783159:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5931, 5931] 00000080:00200000:1.0:1789760750.783245:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.783257:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.783286:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24297472 i_blocks 49159 00000020:00200000:1.0:1789760750.783293:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.783315:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.783365:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24297472, count: 4096 00000080:00200000:1.0:1789760750.783376:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24297472, bytes: 4096 00000080:00200000:1.0:1789760750.783399:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.783412:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.783414:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24297472, 24301568) 0 0 00000080:00200000:1.0:1789760750.783432:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.783458:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5932, 5932] 00020000:00200000:1.0:1789760750.783472:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24297472 chunk: [24297472, 24301568) 24301568, 4096 00020000:00200000:1.0:1789760750.783486:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.783520:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5932, 5932] 00000008:00010000:1.0:1789760750.783554:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.783563:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.783579:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24297472, 24301568) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.783599:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24297472=0x172c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.783754:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.783760:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24301568 00000020:00010000:1.0:1789760750.783773:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.783779:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.783782:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.783788:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.783794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.783807:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.783815:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.783852:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760750.784145:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 24297472<>24301568 00000100:00100000:0.0:1789760750.784799:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760750.784827:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840384:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.834887:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973249 gen 2 00000008:00010000:0.0:1789760750.834954:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760750.834964:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.834973:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.834995:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760750.835028:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840384:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.835620:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5932, 5932] 00000080:00200000:1.0:1789760750.835641:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.835648:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.835665:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24301568 i_blocks 49159 00000020:00200000:1.0:1789760750.835669:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.835681:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.835716:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24301568, count: 4096 00000080:00200000:1.0:1789760750.835724:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24301568, bytes: 4096 00000080:00200000:1.0:1789760750.835736:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.835745:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.835748:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24301568, 24305664) 0 0 00000080:00200000:1.0:1789760750.835753:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.835771:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5933, 5933] 00020000:00200000:1.0:1789760750.835780:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24301568 chunk: [24301568, 24305664) 24305664, 4096 00020000:00200000:1.0:1789760750.835785:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.835806:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5933, 5933] 00000008:00010000:1.0:1789760750.835826:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.835832:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.835845:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24301568, 24305664) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.835870:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24301568=0x172d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.835932:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.835935:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24305664 00000020:00010000:1.0:1789760750.835943:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.835947:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.835950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.835954:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.835957:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.835965:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.835971:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.835989:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760750.836117:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 24301568<>24305664 00000100:00100000:0.0:1789760750.836554:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840512:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.876069:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973250 gen 2 00000008:00010000:0.0:1789760750.876117:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760750.876125:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.876136:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.876158:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760750.876210:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840512:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.876310:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5933, 5933] 00000080:00200000:1.0:1789760750.876328:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.876334:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.876349:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24305664 i_blocks 49159 00000020:00200000:1.0:1789760750.876353:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.876362:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.876397:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24305664, count: 4096 00000080:00200000:1.0:1789760750.876405:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24305664, bytes: 4096 00000080:00200000:1.0:1789760750.876413:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.876422:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.876424:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24305664, 24309760) 0 0 00000080:00200000:1.0:1789760750.876429:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.876446:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5934, 5934] 00020000:00200000:1.0:1789760750.876454:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24305664 chunk: [24305664, 24309760) 24309760, 4096 00020000:00200000:1.0:1789760750.876459:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.876479:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5934, 5934] 00000008:00010000:1.0:1789760750.876496:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.876502:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.876513:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24305664, 24309760) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.876526:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24305664=0x172e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.876721:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.876725:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24309760 00000020:00010000:1.0:1789760750.876734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.876738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.876741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.876744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.876747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.876755:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.876761:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.876778:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760750.876909:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 24305664<>24309760 00000100:00100000:0.0:1789760750.876978:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495840640:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.916247:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973251 gen 2 00000008:00010000:0.0:1789760750.916312:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760750.916321:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.916333:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.916354:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760750.916384:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495840640:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.916616:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5934, 5934] 00000080:00200000:1.0:1789760750.916632:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.916640:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.916655:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24309760 i_blocks 49159 00000020:00200000:1.0:1789760750.916658:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.916667:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.916699:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24309760, count: 4096 00000080:00200000:1.0:1789760750.916707:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24309760, bytes: 4096 00000080:00200000:1.0:1789760750.916717:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.916726:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.916729:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24309760, 24313856) 0 0 00000080:00200000:1.0:1789760750.916733:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.916749:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5935, 5935] 00020000:00200000:1.0:1789760750.916758:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24309760 chunk: [24309760, 24313856) 24313856, 4096 00020000:00200000:1.0:1789760750.916763:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.916783:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5935, 5935] 00000008:00010000:1.0:1789760750.916800:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.916806:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.916819:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24309760, 24313856) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.916833:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24309760=0x172f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.916903:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.916906:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24313856 00000020:00010000:1.0:1789760750.916914:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.916918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.916921:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.916925:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.916928:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.916936:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.916942:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.916959:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760750.917088:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 24309760<>24313856 00000100:00100000:0.0:1789760750.917277:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840768:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760750.974489:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973252 gen 2 00000008:00010000:0.0:1789760750.974550:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760750.974560:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760750.974573:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760750.974598:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760750.974628:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840768:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760750.974743:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5935, 5935] 00000080:00200000:1.0:1789760750.974760:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.974766:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.974781:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24313856 i_blocks 49159 00000020:00200000:1.0:1789760750.974785:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.974795:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.974828:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24313856, count: 4096 00000080:00200000:1.0:1789760750.974835:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24313856, bytes: 4096 00000080:00200000:1.0:1789760750.974849:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.974858:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.974860:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24313856, 24317952) 0 0 00000080:00200000:1.0:1789760750.974865:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.974881:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5936, 5936] 00020000:00200000:1.0:1789760750.974890:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24313856 chunk: [24313856, 24317952) 24317952, 4096 00020000:00200000:1.0:1789760750.974895:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.974913:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5936, 5936] 00000008:00010000:1.0:1789760750.974931:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.974937:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760750.974948:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24313856, 24317952) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.974962:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24313856=0x1730000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.975024:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.975027:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24317952 00000020:00010000:1.0:1789760750.975036:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.975040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.975043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.975047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.975050:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.975057:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.975064:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.975080:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760750.975237:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 24313856<>24317952 00000100:00100000:0.0:1789760750.975390:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760750.975408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840896:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.032675:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973253 gen 2 00000008:00010000:0.0:1789760751.033647:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760751.033658:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.033670:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.033696:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000080:00200000:1.0:1789760751.034916:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5936, 5936] 00000100:00100000:0.0:1789760751.034933:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495840896:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.034989:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.034996:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.035012:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24317952 i_blocks 49159 00000020:00200000:1.0:1789760751.035017:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.035029:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.035064:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24317952, count: 4096 00000080:00200000:1.0:1789760751.035071:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24317952, bytes: 4096 00000080:00200000:1.0:1789760751.035085:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.035093:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.035095:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24317952, 24322048) 0 0 00000080:00200000:1.0:1789760751.035100:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.035118:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5937, 5937] 00020000:00200000:1.0:1789760751.035128:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24317952 chunk: [24317952, 24322048) 24322048, 4096 00020000:00200000:1.0:1789760751.035132:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.035154:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5937, 5937] 00000008:00010000:1.0:1789760751.035195:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.035200:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.035212:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24317952, 24322048) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.035228:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24317952=0x1731000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.035281:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.035284:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24322048 00000020:00010000:1.0:1789760751.035292:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.035296:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.035299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.035302:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.035305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.035312:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.035318:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.035334:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760751.035466:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 24317952<>24322048 00000100:00100000:0.0:1789760751.035844:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760751.035990:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495841024:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760751.051770:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760751.068030:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973254 gen 2 00000008:00010000:0.0:1789760751.068076:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760751.068085:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.068095:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.068112:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760751.068139:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495841024:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.068252:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5937, 5937] 00000080:00200000:1.0:1789760751.068268:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.068274:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.068290:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24322048 i_blocks 49159 00000020:00200000:1.0:1789760751.068293:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.068303:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.068335:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24322048, count: 4096 00000080:00200000:1.0:1789760751.068342:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24322048, bytes: 4096 00000080:00200000:1.0:1789760751.068351:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.068360:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.068363:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24322048, 24326144) 0 0 00000080:00200000:1.0:1789760751.068367:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.068383:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5938, 5938] 00020000:00200000:1.0:1789760751.068393:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24322048 chunk: [24322048, 24326144) 24326144, 4096 00020000:00200000:1.0:1789760751.068398:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.068416:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5938, 5938] 00000008:00010000:1.0:1789760751.068434:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.068439:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.068450:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24322048, 24326144) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.068470:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24322048=0x1732000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.068599:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.068602:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24326144 00000020:00010000:1.0:1789760751.068611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.068614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.068617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.068621:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.068623:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.068631:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.068638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.068654:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760751.068776:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 24322048<>24326144 00000100:00100000:0.0:1789760751.068838:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495841152:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.099153:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973255 gen 2 00000008:00010000:0.0:1789760751.099236:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760751.099245:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.099255:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.099318:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760751.099357:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495841152:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.099489:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5938, 5938] 00000080:00200000:1.0:1789760751.099505:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.099511:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.099588:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24326144 i_blocks 49159 00000020:00200000:1.0:1789760751.099591:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.099600:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.099634:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24326144, count: 4096 00000080:00200000:1.0:1789760751.099641:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24326144, bytes: 4096 00000080:00200000:1.0:1789760751.099650:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.099659:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.099661:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24326144, 24330240) 0 0 00000080:00200000:1.0:1789760751.099665:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.099681:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5939, 5939] 00020000:00200000:1.0:1789760751.099690:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24326144 chunk: [24326144, 24330240) 24330240, 4096 00020000:00200000:1.0:1789760751.099695:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.099714:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5939, 5939] 00000008:00010000:1.0:1789760751.099732:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.099738:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.099750:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24326144, 24330240) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.099764:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24326144=0x1733000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.099822:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.099825:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24330240 00000020:00010000:1.0:1789760751.099833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.099838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.099840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.099844:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.099981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.099993:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.100000:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.100016:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760751.100140:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 24326144<>24330240 00000100:00100000:0.0:1789760751.100213:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495841280:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.134806:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973256 gen 2 00000008:00010000:0.0:1789760751.134874:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760751.134883:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.134894:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.134915:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760751.134944:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495841280:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.135049:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5939, 5939] 00000080:00200000:1.0:1789760751.135066:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.135073:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.135089:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24330240 i_blocks 49159 00000020:00200000:1.0:1789760751.135093:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.135102:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.135136:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24330240, count: 4096 00000080:00200000:1.0:1789760751.135144:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24330240, bytes: 4096 00000080:00200000:1.0:1789760751.135154:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.135163:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.135166:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24330240, 24334336) 0 0 00000080:00200000:1.0:1789760751.135202:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.135221:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5940, 5940] 00020000:00200000:1.0:1789760751.135230:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24330240 chunk: [24330240, 24334336) 24334336, 4096 00020000:00200000:1.0:1789760751.135235:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.135255:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5940, 5940] 00000008:00010000:1.0:1789760751.135272:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.135278:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.135290:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24330240, 24334336) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.135304:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24330240=0x1734000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.135367:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.135369:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24334336 00000020:00010000:1.0:1789760751.135378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.135381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.135384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.135388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.135391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.135398:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.135405:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.135422:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760751.135549:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 24330240<>24334336 00000100:00100000:0.0:1789760751.136023:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495841408:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.181113:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973257 gen 2 00000008:00010000:0.0:1789760751.181160:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760751.181205:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.181218:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.181238:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760751.181266:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495841408:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.181521:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5940, 5940] 00000080:00200000:1.0:1789760751.181546:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.181553:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.181568:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24334336 i_blocks 49159 00000020:00200000:1.0:1789760751.181572:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.181581:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.181614:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24334336, count: 4096 00000080:00200000:1.0:1789760751.181621:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24334336, bytes: 4096 00000080:00200000:1.0:1789760751.181630:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.181638:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.181640:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24334336, 24338432) 0 0 00000080:00200000:1.0:1789760751.181644:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.181661:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5941, 5941] 00020000:00200000:1.0:1789760751.181670:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24334336 chunk: [24334336, 24338432) 24338432, 4096 00020000:00200000:1.0:1789760751.181674:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.181694:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5941, 5941] 00000008:00010000:1.0:1789760751.181713:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.181719:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.181731:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24334336, 24338432) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.181745:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24334336=0x1735000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.181802:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.181804:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24338432 00000020:00010000:1.0:1789760751.181814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.181817:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.181820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.181824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.181826:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.181833:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.181840:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.181871:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760751.182003:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 24334336<>24338432 00000100:00100000:0.0:1789760751.182706:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495841536:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760751.223455:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760751.223476:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.223492:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.223629:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760751.223660:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495841536:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.223742:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5941, 5941] 00000080:00200000:1.0:1789760751.223761:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.223769:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.223789:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24338432 i_blocks 49159 00000020:00200000:1.0:1789760751.223793:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.223805:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.223914:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24338432, count: 4096 00000080:00200000:1.0:1789760751.223923:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24338432, bytes: 4096 00000080:00200000:1.0:1789760751.223935:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.223944:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.223947:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24338432, 24342528) 0 0 00000080:00200000:1.0:1789760751.223952:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.223970:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5942, 5942] 00020000:00200000:1.0:1789760751.223980:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24338432 chunk: [24338432, 24342528) 24342528, 4096 00020000:00200000:1.0:1789760751.223986:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.224011:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5942, 5942] 00000008:00010000:1.0:1789760751.224031:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.224038:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.224051:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24338432, 24342528) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.224068:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24338432=0x1736000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.224138:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.224141:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24342528 00000020:00010000:1.0:1789760751.224152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.224156:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.224159:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.224164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.225320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.225407:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.225419:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.225455:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760751.225663:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 24338432<>24342528 00000100:00100000:0.0:1789760751.225749:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495841664:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.273266:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973258 gen 2 00000100:00100000:0.0:1789760751.273307:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504c380 x1876699495841664/t4294973259(4294973259) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760767 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760751.273356:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760751.273363:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.273374:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.273392:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760751.273416:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495841664:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.273569:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5942, 5942] 00000080:00200000:1.0:1789760751.273586:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.273593:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.273609:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24342528 i_blocks 49159 00000020:00200000:1.0:1789760751.273612:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.273621:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.273656:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24342528, count: 4096 00000080:00200000:1.0:1789760751.273664:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24342528, bytes: 4096 00000080:00200000:1.0:1789760751.273675:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.273685:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.273687:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24342528, 24346624) 0 0 00000080:00200000:1.0:1789760751.273692:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.273709:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5943, 5943] 00020000:00200000:1.0:1789760751.273717:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24342528 chunk: [24342528, 24346624) 24346624, 4096 00020000:00200000:1.0:1789760751.273722:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.273741:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5943, 5943] 00000008:00010000:1.0:1789760751.273758:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.273763:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.273774:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24342528, 24346624) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.273788:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24342528=0x1737000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.273839:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.273842:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24346624 00000020:00010000:1.0:1789760751.273850:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.273853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.273856:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.273859:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.273862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.273869:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.273875:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.273890:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760751.274012:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 24342528<>24346624 00000100:00100000:0.0:1789760751.274325:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495841792:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.305309:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973260 gen 2 00000008:00010000:0.0:1789760751.305376:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760751.305386:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.305398:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.305419:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760751.305451:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495841792:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.305743:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5943, 5943] 00000080:00200000:1.0:1789760751.305760:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.305766:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.305782:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24346624 i_blocks 49159 00000020:00200000:1.0:1789760751.305786:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.305796:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.305832:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24346624, count: 4096 00000080:00200000:1.0:1789760751.305839:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24346624, bytes: 4096 00000080:00200000:1.0:1789760751.305850:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.305993:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.305996:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24346624, 24350720) 0 0 00000080:00200000:1.0:1789760751.306001:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.306087:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5944, 5944] 00020000:00200000:1.0:1789760751.306096:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24346624 chunk: [24346624, 24350720) 24350720, 4096 00020000:00200000:1.0:1789760751.306101:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.306121:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5944, 5944] 00000008:00010000:1.0:1789760751.306139:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.306145:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.306158:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24346624, 24350720) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.306612:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24346624=0x1738000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.306684:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.306687:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24350720 00000020:00010000:1.0:1789760751.306698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.306702:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.306705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.306708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.306711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.306720:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.306727:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.306748:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760751.307608:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 24346624<>24350720 00000100:00100000:1.0:1789760751.307668:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760751.307692:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495841920:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.371989:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973261 gen 2 00000008:00010000:0.0:1789760751.372283:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760751.372293:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.372304:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.372401:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760751.372428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495841920:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760751.376113:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5944, 5944] 00000080:00200000:2.0:1789760751.376142:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.376150:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.376185:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24350720 i_blocks 49159 00000020:00200000:2.0:1789760751.376263:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.376281:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.376316:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24350720, count: 4096 00000080:00200000:2.0:1789760751.376325:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24350720, bytes: 4096 00000080:00200000:2.0:1789760751.376341:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.376350:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.376352:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24350720, 24354816) 0 0 00000080:00200000:2.0:1789760751.376357:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.376374:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5945, 5945] 00020000:00200000:2.0:1789760751.376384:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24350720 chunk: [24350720, 24354816) 24354816, 4096 00020000:00200000:2.0:1789760751.376388:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.376410:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5945, 5945] 00000008:00010000:2.0:1789760751.376429:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.376434:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760751.376446:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24350720, 24354816) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.376460:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24350720=0x1739000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.376528:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.376531:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24354816 00000020:00010000:2.0:1789760751.376552:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.376556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.376560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.376563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.376567:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.376575:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.376582:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.376614:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760751.376747:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 24350720<>24354816 00000100:00100000:3.0:1789760751.376851:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495842048:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760751.410147:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973262 gen 2 00000008:00010000:3.0:1789760751.410222:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760751.410230:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.410241:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.410326:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760751.410357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495842048:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760751.418426:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5945, 5945] 00000080:00200000:0.0:1789760751.418457:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760751.418464:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760751.418486:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24354816 i_blocks 49159 00000020:00200000:0.0:1789760751.418492:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760751.418511:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760751.418565:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24354816, count: 4096 00000080:00200000:0.0:1789760751.418574:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24354816, bytes: 4096 00000080:00200000:0.0:1789760751.418586:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760751.418596:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760751.418599:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24354816, 24358912) 0 0 00000080:00200000:0.0:1789760751.418604:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760751.418622:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5946, 5946] 00020000:00200000:0.0:1789760751.418630:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24354816 chunk: [24354816, 24358912) 24358912, 4096 00020000:00200000:0.0:1789760751.418635:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760751.418985:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5946, 5946] 00000008:00010000:0.0:1789760751.419008:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760751.419013:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760751.419026:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24354816, 24358912) append 0 append lockpos 0 00000080:00200000:0.0:1789760751.419040:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24354816=0x173a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760751.419125:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760751.419128:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24358912 00000020:00010000:0.0:1789760751.419136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760751.419140:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760751.419143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760751.419147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760751.419150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760751.419159:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760751.419166:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760751.419236:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760751.419386:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 24354816<>24358912 00000100:00100000:0.0:1789760751.419455:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495842176:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760751.451043:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760751.451060:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.451071:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.451092:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760751.451122:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495842176:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.451300:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5946, 5946] 00000080:00200000:1.0:1789760751.451317:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.451324:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.451339:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24358912 i_blocks 49159 00000020:00200000:1.0:1789760751.451343:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.451361:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.451406:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24358912, count: 4096 00000080:00200000:1.0:1789760751.451414:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24358912, bytes: 4096 00000080:00200000:1.0:1789760751.451424:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.451434:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.451437:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24358912, 24363008) 0 0 00000080:00200000:1.0:1789760751.451441:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.451458:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5947, 5947] 00020000:00200000:1.0:1789760751.451467:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24358912 chunk: [24358912, 24363008) 24363008, 4096 00020000:00200000:1.0:1789760751.451471:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.451490:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5947, 5947] 00000008:00010000:1.0:1789760751.451507:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.451513:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.451523:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24358912, 24363008) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.451548:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24358912=0x173b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.451600:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.451603:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24363008 00000020:00010000:1.0:1789760751.451612:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.451615:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.451619:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.451622:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.451626:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.451632:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.451640:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760751.452321:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:0.0:1789760751.452470:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24358912<>24363008 00000100:00100000:0.0:1789760751.452513:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760751.452545:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495842304:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.492121:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973264 gen 2 00000008:00010000:0.0:1789760751.492313:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760751.492331:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.492342:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.492434:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760751.492468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495842304:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760751.492593:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5947, 5947] 00000080:00200000:1.0:1789760751.492610:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.492617:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.492634:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24363008 i_blocks 49159 00000020:00200000:1.0:1789760751.492638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.492647:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.492694:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24363008, count: 4096 00000080:00200000:1.0:1789760751.492702:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24363008, bytes: 4096 00000080:00200000:1.0:1789760751.492717:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.492727:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.492730:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24363008, 24367104) 0 0 00000080:00200000:1.0:1789760751.492734:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.492751:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5948, 5948] 00020000:00200000:1.0:1789760751.492759:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24363008 chunk: [24363008, 24367104) 24367104, 4096 00020000:00200000:1.0:1789760751.492764:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.492783:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5948, 5948] 00000008:00010000:1.0:1789760751.492799:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.492805:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760751.492816:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24363008, 24367104) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.492829:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24363008=0x173c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.492879:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.492882:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24367104 00000020:00010000:1.0:1789760751.492890:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.492894:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.492897:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.492900:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.492903:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.492910:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.492916:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.492931:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760751.493999:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 24363008<>24367104 00000100:00100000:1.0:1789760751.495499:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760751.495544:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495842432:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.534041:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973265 gen 2 00000008:00010000:0.0:1789760751.534094:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760751.534103:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.534114:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.534142:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760751.534203:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495842432:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760751.534553:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5948, 5948] 00000080:00200000:2.0:1789760751.534571:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.534578:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.534594:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24367104 i_blocks 49159 00000020:00200000:2.0:1789760751.534598:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.534608:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.534641:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24367104, count: 4096 00000080:00200000:2.0:1789760751.534648:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24367104, bytes: 4096 00000080:00200000:2.0:1789760751.534660:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.534669:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.534672:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24367104, 24371200) 0 0 00000080:00200000:2.0:1789760751.534676:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.534694:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5949, 5949] 00020000:00200000:2.0:1789760751.534703:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24367104 chunk: [24367104, 24371200) 24371200, 4096 00020000:00200000:2.0:1789760751.534708:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.534727:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5949, 5949] 00000008:00010000:2.0:1789760751.534744:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.534751:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760751.534763:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24367104, 24371200) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.534777:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24367104=0x173d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.534834:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.534837:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24371200 00000020:00010000:2.0:1789760751.534846:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.535010:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.535014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.535018:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.535021:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.535030:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.535037:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.535053:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760751.535195:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 24367104<>24371200 00000100:00100000:3.0:1789760751.535367:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495842560:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760751.569641:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760751.569658:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.569670:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.569693:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760751.569730:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495842560:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760751.572232:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5949, 5949] 00000080:00200000:2.0:1789760751.572263:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.572270:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.572287:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24371200 i_blocks 49159 00000020:00200000:2.0:1789760751.572293:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.572309:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.572343:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24371200, count: 4096 00000080:00200000:2.0:1789760751.572351:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24371200, bytes: 4096 00000080:00200000:2.0:1789760751.572366:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.572375:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.572377:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24371200, 24375296) 0 0 00000080:00200000:2.0:1789760751.572381:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.572396:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5950, 5950] 00020000:00200000:2.0:1789760751.572404:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24371200 chunk: [24371200, 24375296) 24375296, 4096 00020000:00200000:2.0:1789760751.572409:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.572429:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5950, 5950] 00000008:00010000:2.0:1789760751.572449:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.572454:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760751.572466:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24371200, 24375296) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.572480:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24371200=0x173e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.572572:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.572575:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24375296 00000020:00010000:2.0:1789760751.572584:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.572587:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.572590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.572593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.572596:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.572603:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.572609:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.572630:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760751.572774:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 24371200<>24375296 00000100:00100000:3.0:1789760751.573225:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760751.573252:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495842688:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760751.632682:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973266 gen 2 00000100:00100000:3.0:1789760751.632727:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495842688/t4294973267(4294973267) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760767 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760751.632775:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760751.632783:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.632793:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.632938:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760751.632965:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495842688:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760751.633040:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5950, 5950] 00000080:00200000:2.0:1789760751.633057:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.633064:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.633081:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24375296 i_blocks 49159 00000020:00200000:2.0:1789760751.633084:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.633095:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.633133:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24375296, count: 4096 00000080:00200000:2.0:1789760751.633140:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24375296, bytes: 4096 00000080:00200000:2.0:1789760751.633151:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.633160:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.633163:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24375296, 24379392) 0 0 00000080:00200000:2.0:1789760751.633204:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.633225:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5951, 5951] 00020000:00200000:2.0:1789760751.633233:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24375296 chunk: [24375296, 24379392) 24379392, 4096 00020000:00200000:2.0:1789760751.633238:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.633271:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5951, 5951] 00000008:00010000:2.0:1789760751.635275:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.635298:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760751.635324:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24375296, 24379392) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.635344:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24375296=0x173f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.635424:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.635428:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24379392 00000020:00010000:2.0:1789760751.635438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.635443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.635446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.635450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.635453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.635465:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.635474:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.635507:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760751.635651:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 24375296<>24379392 00000100:00100000:3.0:1789760751.635947:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760751.635974:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495842816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760751.685329:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973268 gen 2 00000008:00010000:3.0:1789760751.685392:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760751.685401:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.685411:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.685435:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760751.685465:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495842816:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760751.685631:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5951, 5951] 00000080:00200000:2.0:1789760751.685648:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.685654:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.685669:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24379392 i_blocks 49159 00000020:00200000:2.0:1789760751.685672:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.685681:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.685715:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24379392, count: 4096 00000080:00200000:2.0:1789760751.685721:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24379392, bytes: 4096 00000080:00200000:2.0:1789760751.685731:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.685739:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.685741:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24379392, 24383488) 0 0 00000080:00200000:2.0:1789760751.685745:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.685761:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5952, 5952] 00020000:00200000:2.0:1789760751.685768:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24379392 chunk: [24379392, 24383488) 24383488, 4096 00020000:00200000:2.0:1789760751.685772:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.685790:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5952, 5952] 00000008:00010000:2.0:1789760751.685808:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.685814:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760751.685825:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24379392, 24383488) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.685837:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24379392=0x1740000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.685952:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.685956:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24383488 00000020:00010000:2.0:1789760751.685965:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.685969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.685971:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.685975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.685978:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.685985:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.685991:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.686005:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760751.686122:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 24379392<>24383488 00000100:00100000:3.0:1789760751.686240:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760751.686259:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495842944:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760751.720697:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973269 gen 2 00000008:00010000:3.0:1789760751.720745:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760751.720753:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.720764:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.720788:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760751.720817:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495842944:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760751.720841:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5952, 5952] 00000080:00200000:2.0:1789760751.720914:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.720921:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.720937:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24383488 i_blocks 49159 00000020:00200000:2.0:1789760751.720941:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.720950:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.720985:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24383488, count: 4096 00000080:00200000:2.0:1789760751.720992:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24383488, bytes: 4096 00000080:00200000:2.0:1789760751.721004:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.721013:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.721015:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24383488, 24387584) 0 0 00000080:00200000:2.0:1789760751.721019:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.721036:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5953, 5953] 00020000:00200000:2.0:1789760751.721044:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24383488 chunk: [24383488, 24387584) 24387584, 4096 00020000:00200000:2.0:1789760751.721049:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.721069:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5953, 5953] 00000008:00010000:2.0:1789760751.721087:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.721093:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760751.721104:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24383488, 24387584) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.721118:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24383488=0x1741000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.721205:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.721209:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24387584 00000020:00010000:2.0:1789760751.721218:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.721220:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.721223:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.721225:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.721227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.721232:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.721238:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.721251:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760751.721562:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 24383488<>24387584 00000100:00100000:2.0:1789760751.721611:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760751.721628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495843072:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760751.789131:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973270 gen 2 00000008:00010000:2.0:1789760751.789192:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760751.789200:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.789209:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.789232:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760751.789254:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495843072:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760751.789355:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5953, 5953] 00000080:00200000:3.0:1789760751.789393:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.789399:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.789411:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24387584 i_blocks 49159 00000020:00200000:3.0:1789760751.789415:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.789422:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.789468:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24387584, count: 4096 00000080:00200000:3.0:1789760751.789475:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24387584, bytes: 4096 00000080:00200000:3.0:1789760751.789482:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.789489:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.789492:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24387584, 24391680) 0 0 00000080:00200000:3.0:1789760751.789496:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.789509:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5954, 5954] 00020000:00200000:3.0:1789760751.789516:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24387584 chunk: [24387584, 24391680) 24391680, 4096 00020000:00200000:3.0:1789760751.789520:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.789548:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5954, 5954] 00000008:00010000:3.0:1789760751.789562:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.789567:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760751.789576:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24387584, 24391680) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.789587:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24387584=0x1742000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.789632:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.789635:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24391680 00000020:00010000:3.0:1789760751.789643:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.789647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.789649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.789652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.789655:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.789661:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.789667:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.789680:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760751.789776:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 24387584<>24391680 00000100:00100000:3.0:1789760751.789817:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760751.789832:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495843200:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760751.831028:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:3.0:1789760751.831045:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.831056:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.831077:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:3.0:1789760751.831105:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495843200:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760751.831148:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5954, 5954] 00000080:00200000:0.0:1789760751.831201:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760751.831207:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760751.831222:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24391680 i_blocks 49159 00000020:00200000:0.0:1789760751.831227:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760751.831238:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760751.831265:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24391680, count: 4096 00000080:00200000:0.0:1789760751.831271:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24391680, bytes: 4096 00000080:00200000:0.0:1789760751.831281:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760751.831288:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760751.831291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24391680, 24395776) 0 0 00000080:00200000:0.0:1789760751.831294:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760751.831307:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5955, 5955] 00020000:00200000:0.0:1789760751.831314:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24391680 chunk: [24391680, 24395776) 24395776, 4096 00020000:00200000:0.0:1789760751.831318:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760751.831335:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5955, 5955] 00000008:00010000:0.0:1789760751.831351:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760751.831356:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760751.831365:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24391680, 24395776) append 0 append lockpos 0 00000080:00200000:0.0:1789760751.831376:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24391680=0x1743000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760751.831428:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760751.831430:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24395776 00000020:00010000:0.0:1789760751.831438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760751.831441:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760751.831443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760751.831446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760751.831449:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760751.831455:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760751.831461:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760751.831480:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760751.831592:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24391680<>24395776 00000100:00100000:1.0:1789760751.831680:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495843328:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760751.860107:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973272 gen 2 00000008:00010000:0.0:1789760751.860220:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760751.860229:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760751.860240:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760751.860265:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760751.860293:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495843328:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760751.860392:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5955, 5955] 00000080:00200000:3.0:1789760751.860409:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.860421:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.860436:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24395776 i_blocks 49159 00000020:00200000:3.0:1789760751.860439:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.860449:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.860482:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24395776, count: 4096 00000080:00200000:3.0:1789760751.860489:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24395776, bytes: 4096 00000080:00200000:3.0:1789760751.860499:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.860507:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.860519:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24395776, 24399872) 0 0 00000080:00200000:3.0:1789760751.860524:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.860551:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5956, 5956] 00020000:00200000:3.0:1789760751.860559:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24395776 chunk: [24395776, 24399872) 24399872, 4096 00020000:00200000:3.0:1789760751.860564:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.860585:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5956, 5956] 00000008:00010000:3.0:1789760751.860602:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.860607:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760751.860620:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24395776, 24399872) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.860634:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24395776=0x1744000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.860690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.860692:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24399872 00000020:00010000:3.0:1789760751.860700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.860703:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.860706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.860709:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.860712:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.860718:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.860724:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.860739:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760751.860854:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 24395776<>24399872 00000100:00100000:2.0:1789760751.861045:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495843456:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760751.910983:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760751.911005:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.911016:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.911035:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760751.911060:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495843456:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760751.911103:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5956, 5956] 00000080:00200000:3.0:1789760751.911127:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.911133:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.911150:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24399872 i_blocks 49159 00000020:00200000:3.0:1789760751.911155:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.911166:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.911231:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24399872, count: 4096 00000080:00200000:3.0:1789760751.911239:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24399872, bytes: 4096 00000080:00200000:3.0:1789760751.911247:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.911315:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.911317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24399872, 24403968) 0 0 00000080:00200000:3.0:1789760751.911322:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.911339:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5957, 5957] 00020000:00200000:3.0:1789760751.911347:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24399872 chunk: [24399872, 24403968) 24403968, 4096 00020000:00200000:3.0:1789760751.911351:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.911373:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5957, 5957] 00000008:00010000:3.0:1789760751.911392:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.911398:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760751.911421:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24399872, 24403968) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.911435:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24399872=0x1745000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.911493:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.911497:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24403968 00000020:00010000:3.0:1789760751.911668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.911673:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.911677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.911681:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.911684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.911694:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.911701:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.911721:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760751.911973:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 24399872<>24403968 00000100:00100000:2.0:1789760751.912303:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495843584:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760751.954062:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973273 gen 2 00000100:00100000:2.0:1789760751.954105:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203145180 x1876699495843584/t4294973274(4294973274) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760767 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760751.954494:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760751.954502:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.954517:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.954548:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760751.954575:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495843584:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760751.954585:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5957, 5957] 00000080:00200000:3.0:1789760751.954605:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.954613:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.954629:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24403968 i_blocks 49159 00000020:00200000:3.0:1789760751.954633:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.954643:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.954678:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24403968, count: 4096 00000080:00200000:3.0:1789760751.954685:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24403968, bytes: 4096 00000080:00200000:3.0:1789760751.954695:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.954703:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.954706:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24403968, 24408064) 0 0 00000080:00200000:3.0:1789760751.954710:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.954727:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5958, 5958] 00020000:00200000:3.0:1789760751.954735:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24403968 chunk: [24403968, 24408064) 24408064, 4096 00020000:00200000:3.0:1789760751.954740:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.954760:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5958, 5958] 00000008:00010000:3.0:1789760751.954779:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.954785:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760751.954797:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24403968, 24408064) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.954812:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24403968=0x1746000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.955154:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.955158:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24408064 00000020:00010000:3.0:1789760751.955204:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.955209:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.955212:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.955215:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.955219:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.955227:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.955235:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.955254:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760751.955384:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 24403968<>24408064 00000100:00100000:3.0:1789760751.955436:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760751.955453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495843712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.001046:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973275 gen 2 00000008:00010000:3.0:1789760752.001120:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:3.0:1789760752.001130:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.001142:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.001201:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:3.0:1789760752.001231:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495843712:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.001340:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5958, 5958] 00000080:00200000:2.0:1789760752.001358:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.001365:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.001381:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24408064 i_blocks 49159 00000020:00200000:2.0:1789760752.001384:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.001393:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.001427:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24408064, count: 4096 00000080:00200000:2.0:1789760752.001434:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24408064, bytes: 4096 00000080:00200000:2.0:1789760752.001444:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.001453:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.001455:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24408064, 24412160) 0 0 00000080:00200000:2.0:1789760752.001460:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.001477:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5959, 5959] 00020000:00200000:2.0:1789760752.001485:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24408064 chunk: [24408064, 24412160) 24412160, 4096 00020000:00200000:2.0:1789760752.001490:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.001510:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5959, 5959] 00000008:00010000:2.0:1789760752.001539:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.001546:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.001557:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24408064, 24412160) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.001573:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24408064=0x1747000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.001630:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.001632:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24412160 00000020:00010000:2.0:1789760752.001640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.001644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.001648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.001651:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.001654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.001662:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.001669:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.001684:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760752.001805:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 24408064<>24412160 00000100:00100000:2.0:1789760752.001846:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760752.001870:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495843840:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760752.033926:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973276 gen 2 00000008:00010000:2.0:1789760752.033982:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760752.033993:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760752.034006:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760752.034040:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760752.034075:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495843840:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760752.034264:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5959, 5959] 00000080:00200000:0.0:1789760752.034282:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760752.034288:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760752.034306:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24412160 i_blocks 49159 00000020:00200000:0.0:1789760752.034310:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760752.034320:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760752.034358:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24412160, count: 4096 00000080:00200000:0.0:1789760752.034371:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24412160, bytes: 4096 00000080:00200000:0.0:1789760752.034382:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760752.034393:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760752.034396:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24412160, 24416256) 0 0 00000080:00200000:0.0:1789760752.034401:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760752.034418:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5960, 5960] 00020000:00200000:0.0:1789760752.034427:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24412160 chunk: [24412160, 24416256) 24416256, 4096 00020000:00200000:0.0:1789760752.034432:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760752.034451:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5960, 5960] 00000008:00010000:0.0:1789760752.034483:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760752.034489:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760752.034500:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24412160, 24416256) append 0 append lockpos 0 00000080:00200000:0.0:1789760752.034514:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24412160=0x1748000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760752.034584:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760752.034587:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24416256 00000020:00010000:0.0:1789760752.034595:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760752.034598:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.034601:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.034605:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760752.034608:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760752.034617:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760752.034624:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760752.034640:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760752.034770:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24412160<>24416256 00000100:00100000:1.0:1789760752.034853:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495843968:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760752.075579:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760752.094752:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973277 gen 2 00000008:00010000:0.0:1789760752.094805:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760752.094812:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760752.094824:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760752.094900:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760752.094930:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495843968:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.099775:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5960, 5960] 00000080:00200000:2.0:1789760752.099807:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.099814:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.099856:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24416256 i_blocks 49159 00000020:00200000:2.0:1789760752.099863:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.099879:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.099911:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24416256, count: 4096 00000080:00200000:2.0:1789760752.099919:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24416256, bytes: 4096 00000080:00200000:2.0:1789760752.099933:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.099942:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.099944:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24416256, 24420352) 0 0 00000080:00200000:2.0:1789760752.099949:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.099966:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5961, 5961] 00020000:00200000:2.0:1789760752.099974:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24416256 chunk: [24416256, 24420352) 24420352, 4096 00020000:00200000:2.0:1789760752.099980:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.100000:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5961, 5961] 00000008:00010000:2.0:1789760752.100018:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.100023:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.100939:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24416256, 24420352) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.100961:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24416256=0x1749000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.101728:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.101734:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24420352 00000020:00010000:2.0:1789760752.101745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.101748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.101753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.101756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.101760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.101773:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.101782:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.101807:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760752.101940:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 24416256<>24420352 00000100:00100000:3.0:1789760752.102277:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844096:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.177993:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973278 gen 2 00000008:00010000:3.0:1789760752.178040:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760752.178048:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.178060:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.178083:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760752.178111:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844096:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.178221:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5961, 5961] 00000080:00200000:2.0:1789760752.178237:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.178242:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.178258:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24420352 i_blocks 49159 00000020:00200000:2.0:1789760752.178261:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.178270:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.178303:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24420352, count: 4096 00000080:00200000:2.0:1789760752.178310:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24420352, bytes: 4096 00000080:00200000:2.0:1789760752.178319:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.178328:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.178330:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24420352, 24424448) 0 0 00000080:00200000:2.0:1789760752.178335:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.178351:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5962, 5962] 00020000:00200000:2.0:1789760752.178359:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24420352 chunk: [24420352, 24424448) 24424448, 4096 00020000:00200000:2.0:1789760752.178364:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.178382:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5962, 5962] 00000008:00010000:2.0:1789760752.178397:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.178412:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.178423:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24420352, 24424448) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.178435:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24420352=0x174a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.178965:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.178970:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24424448 00000020:00010000:2.0:1789760752.178980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.178983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.178986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.178990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.178993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.179002:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.179008:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.179028:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760752.179153:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 24420352<>24424448 00000100:00100000:3.0:1789760752.179620:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495844224:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760752.213808:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760752.213822:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.213834:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.213861:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760752.213890:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495844224:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.213989:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5962, 5962] 00000080:00200000:2.0:1789760752.214073:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.214081:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.214097:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24424448 i_blocks 49159 00000020:00200000:2.0:1789760752.214101:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.214111:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.214144:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24424448, count: 4096 00000080:00200000:2.0:1789760752.214151:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24424448, bytes: 4096 00000080:00200000:2.0:1789760752.214161:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.214252:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.214254:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24424448, 24428544) 0 0 00000080:00200000:2.0:1789760752.214259:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.214275:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5963, 5963] 00020000:00200000:2.0:1789760752.214284:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24424448 chunk: [24424448, 24428544) 24428544, 4096 00020000:00200000:2.0:1789760752.214290:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.214311:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5963, 5963] 00000008:00010000:2.0:1789760752.214330:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.214337:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.214359:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24424448, 24428544) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.214376:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24424448=0x174b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.214431:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.214433:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24428544 00000020:00010000:2.0:1789760752.214442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.214446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.214449:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.214453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.214470:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.214477:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.214484:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.214499:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760752.214632:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 24424448<>24428544 00000100:00100000:3.0:1789760752.215195:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844352:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.255120:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973280 gen 2 00000008:00010000:3.0:1789760752.255228:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760752.255237:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.255248:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.255273:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760752.255302:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844352:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.255681:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5963, 5963] 00000080:00200000:2.0:1789760752.255699:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.255705:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.255722:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24428544 i_blocks 49159 00000020:00200000:2.0:1789760752.255725:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.255735:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.255770:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24428544, count: 4096 00000080:00200000:2.0:1789760752.255778:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24428544, bytes: 4096 00000080:00200000:2.0:1789760752.255788:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.255797:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.255800:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24428544, 24432640) 0 0 00000080:00200000:2.0:1789760752.255805:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.255823:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5964, 5964] 00020000:00200000:2.0:1789760752.255831:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24428544 chunk: [24428544, 24432640) 24432640, 4096 00020000:00200000:2.0:1789760752.255836:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.255950:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5964, 5964] 00000008:00010000:2.0:1789760752.255969:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.255974:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.255986:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24428544, 24432640) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.256001:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24428544=0x174c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.256131:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.256135:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24432640 00000020:00010000:2.0:1789760752.256145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.256148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.256151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.256155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.256158:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.256191:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.256199:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.256216:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760752.256348:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 24428544<>24432640 00000100:00100000:3.0:1789760752.256455:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760752.256474:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844480:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760752.306865:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760752.306883:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.306895:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.306918:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760752.306951:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844480:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.307419:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5964, 5964] 00000080:00200000:2.0:1789760752.307440:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.307447:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.307462:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24432640 i_blocks 49159 00000020:00200000:2.0:1789760752.307466:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.307475:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.307522:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24432640, count: 4096 00000080:00200000:2.0:1789760752.307539:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24432640, bytes: 4096 00000080:00200000:2.0:1789760752.307551:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.307559:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.307561:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24432640, 24436736) 0 0 00000080:00200000:2.0:1789760752.307566:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.307584:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5965, 5965] 00020000:00200000:2.0:1789760752.307593:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24432640 chunk: [24432640, 24436736) 24436736, 4096 00020000:00200000:2.0:1789760752.307599:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.307618:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5965, 5965] 00000008:00010000:2.0:1789760752.307638:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.307644:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.307655:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24432640, 24436736) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.307670:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24432640=0x174d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.307732:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.307735:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24436736 00000020:00010000:2.0:1789760752.307744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.307748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.307751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.307755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.307758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.307766:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.307773:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.307802:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760752.307936:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 24432640<>24436736 00000100:00100000:3.0:1789760752.308378:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844608:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.355920:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973282 gen 2 00000008:00010000:3.0:1789760752.355989:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760752.355998:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.356009:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.356035:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760752.356066:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844608:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.356261:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5965, 5965] 00000080:00200000:2.0:1789760752.356275:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.356280:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.356293:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24436736 i_blocks 49159 00000020:00200000:2.0:1789760752.356296:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.356303:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.356332:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24436736, count: 4096 00000080:00200000:2.0:1789760752.356338:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24436736, bytes: 4096 00000080:00200000:2.0:1789760752.356345:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.356353:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.356354:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24436736, 24440832) 0 0 00000080:00200000:2.0:1789760752.356358:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.356371:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5966, 5966] 00020000:00200000:2.0:1789760752.356377:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24436736 chunk: [24436736, 24440832) 24440832, 4096 00020000:00200000:2.0:1789760752.356381:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.356397:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5966, 5966] 00000008:00010000:2.0:1789760752.356411:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.356416:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.356425:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24436736, 24440832) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.356435:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24436736=0x174e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.356479:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.356481:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24440832 00000020:00010000:2.0:1789760752.356488:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.356491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.356493:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.356496:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.356498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.356502:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.356507:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.356521:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760752.356647:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 24436736<>24440832 00000100:00100000:3.0:1789760752.356750:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495844736:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760752.409944:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760752.409960:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.409971:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.409994:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000080:00200000:2.0:1789760752.410213:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5966, 5966] 00000080:00200000:2.0:1789760752.410233:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.410239:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.410253:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24440832 i_blocks 49159 00000020:00200000:2.0:1789760752.410257:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.410267:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.410300:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24440832, count: 4096 00000080:00200000:2.0:1789760752.410309:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24440832, bytes: 4096 00000080:00200000:2.0:1789760752.410318:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.410327:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.410329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24440832, 24444928) 0 0 00000080:00200000:2.0:1789760752.410334:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.410351:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5967, 5967] 00020000:00200000:2.0:1789760752.410359:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24440832 chunk: [24440832, 24444928) 24444928, 4096 00020000:00200000:2.0:1789760752.410364:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.410384:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5967, 5967] 00000008:00010000:2.0:1789760752.410401:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.410406:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.410417:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24440832, 24444928) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.410430:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24440832=0x174f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.410491:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.410494:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24444928 00000020:00010000:2.0:1789760752.410502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.410506:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.410509:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.410513:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.410516:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.410523:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.410541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.410561:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760752.410679:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 24440832<>24444928 00000100:00100000:2.0:1789760752.410758:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844864:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.411193:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495844736:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760752.442345:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973284 gen 2 00000008:00010000:2.0:1789760752.442413:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760752.442421:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760752.442431:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760752.442457:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760752.442489:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844864:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760752.442657:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5967, 5967] 00000080:00200000:0.0:1789760752.442676:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760752.442683:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760752.442700:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24444928 i_blocks 49159 00000020:00200000:0.0:1789760752.442704:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760752.442714:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760752.442751:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24444928, count: 4096 00000080:00200000:0.0:1789760752.442759:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24444928, bytes: 4096 00000080:00200000:0.0:1789760752.442771:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760752.442781:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760752.442783:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24444928, 24449024) 0 0 00000080:00200000:0.0:1789760752.442788:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760752.442804:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5968, 5968] 00020000:00200000:0.0:1789760752.442812:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24444928 chunk: [24444928, 24449024) 24449024, 4096 00020000:00200000:0.0:1789760752.442816:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760752.442837:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5968, 5968] 00000008:00010000:0.0:1789760752.442870:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760752.442876:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760752.442889:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24444928, 24449024) append 0 append lockpos 0 00000080:00200000:0.0:1789760752.442902:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24444928=0x1750000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760752.442953:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760752.442956:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24449024 00000020:00010000:0.0:1789760752.442964:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760752.442968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.442971:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.442975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760752.442978:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760752.442984:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760752.442992:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.443381:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:2.0:1789760752.443580:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 24444928<>24449024 00000100:00100000:2.0:1789760752.443624:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760752.443642:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844992:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760752.486258:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973285 gen 2 00000008:00010000:2.0:1789760752.486310:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:2.0:1789760752.486319:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760752.486331:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760752.486355:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:2.0:1789760752.486386:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495844992:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760752.486501:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5968, 5968] 00000080:00200000:0.0:1789760752.486592:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760752.486599:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760752.486615:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24449024 i_blocks 49159 00000020:00200000:0.0:1789760752.486619:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760752.486628:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760752.486665:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24449024, count: 4096 00000080:00200000:0.0:1789760752.486672:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24449024, bytes: 4096 00000080:00200000:0.0:1789760752.486683:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760752.486692:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760752.486695:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24449024, 24453120) 0 0 00000080:00200000:0.0:1789760752.486699:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760752.486715:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5969, 5969] 00020000:00200000:0.0:1789760752.486724:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24449024 chunk: [24449024, 24453120) 24453120, 4096 00020000:00200000:0.0:1789760752.486729:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760752.486749:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5969, 5969] 00000008:00010000:0.0:1789760752.486767:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760752.486772:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760752.486784:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24449024, 24453120) append 0 append lockpos 0 00000080:00200000:0.0:1789760752.486798:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24449024=0x1751000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760752.487294:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760752.487299:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24453120 00000020:00010000:0.0:1789760752.487309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760752.487312:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.487316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.487319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760752.487322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760752.487331:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760752.487338:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760752.487359:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760752.487488:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24449024<>24453120 00000100:00100000:1.0:1789760752.487649:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495845120:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760752.527469:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973286 gen 2 00000008:00010000:0.0:1789760752.527522:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760752.527540:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760752.527550:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760752.527573:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760752.527601:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495845120:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.527728:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5969, 5969] 00000080:00200000:2.0:1789760752.527746:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.527752:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.527768:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24453120 i_blocks 49159 00000020:00200000:2.0:1789760752.527771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.527781:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.527815:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24453120, count: 4096 00000080:00200000:2.0:1789760752.527823:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24453120, bytes: 4096 00000080:00200000:2.0:1789760752.527832:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.527841:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.527843:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24453120, 24457216) 0 0 00000080:00200000:2.0:1789760752.527847:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.527863:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5970, 5970] 00020000:00200000:2.0:1789760752.527871:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24453120 chunk: [24453120, 24457216) 24457216, 4096 00020000:00200000:2.0:1789760752.527876:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.527895:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5970, 5970] 00000008:00010000:2.0:1789760752.527912:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.527918:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.527929:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24453120, 24457216) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.527943:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24453120=0x1752000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.528002:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.528079:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24457216 00000020:00010000:2.0:1789760752.528089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.528093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.528097:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.528101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.528105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.528112:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.528119:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.528135:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760752.528815:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 24453120<>24457216 00000100:00100000:3.0:1789760752.528913:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495845248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.577796:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973287 gen 2 00000008:00010000:3.0:1789760752.577860:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760752.577870:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.577882:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.577904:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760752.577934:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495845248:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.578151:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5970, 5970] 00000080:00200000:2.0:1789760752.578200:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.578207:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.578223:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24457216 i_blocks 49159 00000020:00200000:2.0:1789760752.578226:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.578236:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.578268:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24457216, count: 4096 00000080:00200000:2.0:1789760752.578275:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24457216, bytes: 4096 00000080:00200000:2.0:1789760752.578284:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.578292:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.578294:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24457216, 24461312) 0 0 00000080:00200000:2.0:1789760752.578298:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.578314:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5971, 5971] 00020000:00200000:2.0:1789760752.578322:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24457216 chunk: [24457216, 24461312) 24461312, 4096 00020000:00200000:2.0:1789760752.578327:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.578347:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5971, 5971] 00000008:00010000:2.0:1789760752.578365:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.578371:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.578381:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24457216, 24461312) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.578394:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24457216=0x1753000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.579699:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.579706:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24461312 00000020:00010000:2.0:1789760752.579720:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.579724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.579729:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.579733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.579736:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.579748:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.579756:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.579782:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760752.579919:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 24457216<>24461312 00000100:00100000:3.0:1789760752.580243:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495845376:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.630553:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973288 gen 2 00000008:00010000:3.0:1789760752.630604:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760752.630613:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.630625:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.630649:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760752.630680:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495845376:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.630855:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5971, 5971] 00000080:00200000:2.0:1789760752.630873:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.630879:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.630900:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24461312 i_blocks 49159 00000020:00200000:2.0:1789760752.630905:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.630914:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.630947:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24461312, count: 4096 00000080:00200000:2.0:1789760752.630965:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24461312, bytes: 4096 00000080:00200000:2.0:1789760752.630975:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.630983:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.630986:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24461312, 24465408) 0 0 00000080:00200000:2.0:1789760752.630990:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.631016:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5972, 5972] 00020000:00200000:2.0:1789760752.631025:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24461312 chunk: [24461312, 24465408) 24465408, 4096 00020000:00200000:2.0:1789760752.631029:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.631050:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5972, 5972] 00000008:00010000:2.0:1789760752.631076:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.631081:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.631092:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24461312, 24465408) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.631106:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24461312=0x1754000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.631162:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.631166:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24465408 00000020:00010000:2.0:1789760752.631217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.631221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.631225:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.631228:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.631231:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.631237:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.631244:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.631260:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760752.631378:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 24461312<>24465408 00000100:00100000:3.0:1789760752.631533:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495845504:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.662033:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973289 gen 2 00000008:00010000:3.0:1789760752.662090:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760752.662099:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.662110:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.662136:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760752.662217:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495845504:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.662614:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5972, 5972] 00000080:00200000:2.0:1789760752.662631:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.662638:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.662654:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24465408 i_blocks 49159 00000020:00200000:2.0:1789760752.662658:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.662668:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.662703:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24465408, count: 4096 00000080:00200000:2.0:1789760752.662711:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24465408, bytes: 4096 00000080:00200000:2.0:1789760752.662721:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.662730:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.662732:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24465408, 24469504) 0 0 00000080:00200000:2.0:1789760752.662737:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.662754:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5973, 5973] 00020000:00200000:2.0:1789760752.662762:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24465408 chunk: [24465408, 24469504) 24469504, 4096 00020000:00200000:2.0:1789760752.662766:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.662788:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5973, 5973] 00000008:00010000:2.0:1789760752.662807:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.662813:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.662826:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24465408, 24469504) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.662841:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24465408=0x1755000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.663024:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.663027:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24469504 00000020:00010000:2.0:1789760752.663036:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.663040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.663044:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.663047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.663050:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.663058:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.663066:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.663083:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760752.663235:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 24465408<>24469504 00000100:00100000:3.0:1789760752.663359:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495845632:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760752.694156:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760752.694211:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.694224:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.694245:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760752.694275:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495845632:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.694492:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5973, 5973] 00000080:00200000:2.0:1789760752.694519:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.694534:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.694549:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24469504 i_blocks 49159 00000020:00200000:2.0:1789760752.694552:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.694562:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.694595:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24469504, count: 4096 00000080:00200000:2.0:1789760752.694602:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24469504, bytes: 4096 00000080:00200000:2.0:1789760752.694612:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.694621:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.694623:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24469504, 24473600) 0 0 00000080:00200000:2.0:1789760752.694627:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.694645:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5974, 5974] 00020000:00200000:2.0:1789760752.694653:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24469504 chunk: [24469504, 24473600) 24473600, 4096 00020000:00200000:2.0:1789760752.694658:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.694679:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5974, 5974] 00000008:00010000:2.0:1789760752.694697:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.694704:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.694717:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24469504, 24473600) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.694731:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24469504=0x1756000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.694793:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.694796:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24473600 00000020:00010000:2.0:1789760752.694805:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.694808:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.694812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.694816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.694819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.694828:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.694836:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.695003:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760752.695129:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 24469504<>24473600 00000100:00100000:3.0:1789760752.695273:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495845760:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.724909:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973290 gen 2 00000100:00100000:3.0:1789760752.724950:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495845760/t4294973291(4294973291) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760752.725004:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760752.725012:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.725021:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.725046:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760752.725069:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495845760:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.725225:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5974, 5974] 00000080:00200000:2.0:1789760752.725242:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.725248:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.725264:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24473600 i_blocks 49159 00000020:00200000:2.0:1789760752.725268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.725277:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.725314:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24473600, count: 4096 00000080:00200000:2.0:1789760752.725322:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24473600, bytes: 4096 00000080:00200000:2.0:1789760752.725333:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.725341:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.725344:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24473600, 24477696) 0 0 00000080:00200000:2.0:1789760752.725348:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.725365:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5975, 5975] 00020000:00200000:2.0:1789760752.725373:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24473600 chunk: [24473600, 24477696) 24477696, 4096 00020000:00200000:2.0:1789760752.725378:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.725399:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5975, 5975] 00000008:00010000:2.0:1789760752.725418:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.725424:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.725436:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24473600, 24477696) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.725450:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24473600=0x1757000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.725507:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.725510:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24477696 00000020:00010000:2.0:1789760752.725517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.725520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.725534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.725538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.725541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.725549:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.725556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.725572:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760752.725691:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 24473600<>24477696 00000100:00100000:3.0:1789760752.726146:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495845888:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.756910:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973292 gen 2 00000008:00010000:3.0:1789760752.756978:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760752.756986:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.756997:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.757022:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760752.757063:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495845888:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.757203:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5975, 5975] 00000080:00200000:2.0:1789760752.757221:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.757227:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.757244:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24477696 i_blocks 49159 00000020:00200000:2.0:1789760752.757248:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.757258:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.757294:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24477696, count: 4096 00000080:00200000:2.0:1789760752.757302:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24477696, bytes: 4096 00000080:00200000:2.0:1789760752.757313:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.757323:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.757325:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24477696, 24481792) 0 0 00000080:00200000:2.0:1789760752.757330:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.757350:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5976, 5976] 00020000:00200000:2.0:1789760752.757359:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24477696 chunk: [24477696, 24481792) 24481792, 4096 00020000:00200000:2.0:1789760752.757364:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.757385:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5976, 5976] 00000008:00010000:2.0:1789760752.757404:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.757410:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.757421:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24477696, 24481792) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.757436:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24477696=0x1758000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.757495:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.757498:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24481792 00000020:00010000:2.0:1789760752.757517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.757522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.757534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.757538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.757541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.757550:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.757556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.757574:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760752.757701:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 24477696<>24481792 00000100:00100000:3.0:1789760752.757995:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495846016:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.789395:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973293 gen 2 00000008:00010000:3.0:1789760752.789446:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760752.789455:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.789465:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.789488:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760752.789533:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495846016:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.789686:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5976, 5976] 00000080:00200000:2.0:1789760752.789704:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.789710:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.789726:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24481792 i_blocks 49159 00000020:00200000:2.0:1789760752.789730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.789739:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.789774:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24481792, count: 4096 00000080:00200000:2.0:1789760752.789783:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24481792, bytes: 4096 00000080:00200000:2.0:1789760752.789795:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.789803:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.789805:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24481792, 24485888) 0 0 00000080:00200000:2.0:1789760752.789809:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.789828:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5977, 5977] 00020000:00200000:2.0:1789760752.789836:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24481792 chunk: [24481792, 24485888) 24485888, 4096 00020000:00200000:2.0:1789760752.789841:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.789968:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5977, 5977] 00000008:00010000:2.0:1789760752.789987:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.789992:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.790006:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24481792, 24485888) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.790020:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24481792=0x1759000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.790150:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.790153:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24485888 00000020:00010000:2.0:1789760752.790163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.790166:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.790264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.790268:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.790271:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.790279:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.790285:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.790303:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760752.790426:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 24481792<>24485888 00000100:00100000:3.0:1789760752.790488:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495846144:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.820723:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973294 gen 2 00000008:00010000:3.0:1789760752.820776:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760752.820785:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.820798:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.820821:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760752.820858:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495846144:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.821032:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5977, 5977] 00000080:00200000:2.0:1789760752.821118:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.821125:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.821142:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24485888 i_blocks 49159 00000020:00200000:2.0:1789760752.821146:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.821155:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.821220:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24485888, count: 4096 00000080:00200000:2.0:1789760752.821228:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24485888, bytes: 4096 00000080:00200000:2.0:1789760752.821238:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.821247:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.821250:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24485888, 24489984) 0 0 00000080:00200000:2.0:1789760752.821254:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.821271:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5978, 5978] 00020000:00200000:2.0:1789760752.821280:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24485888 chunk: [24485888, 24489984) 24489984, 4096 00020000:00200000:2.0:1789760752.821284:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.821303:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5978, 5978] 00000008:00010000:2.0:1789760752.821321:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.821327:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.821337:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24485888, 24489984) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.821350:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24485888=0x175a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.821405:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.821408:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24489984 00000020:00010000:2.0:1789760752.821417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.821421:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.821425:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.821428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.821431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.821439:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.821447:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.821464:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760752.821659:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 24485888<>24489984 00000100:00100000:3.0:1789760752.821784:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495846272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.859353:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973295 gen 2 00000008:00010000:3.0:1789760752.859404:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760752.859414:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.859426:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.859451:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760752.859482:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495846272:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.859784:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5978, 5978] 00000080:00200000:2.0:1789760752.859803:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.859810:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.859825:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24489984 i_blocks 49159 00000020:00200000:2.0:1789760752.859829:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.859839:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.859950:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24489984, count: 4096 00000080:00200000:2.0:1789760752.859959:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24489984, bytes: 4096 00000080:00200000:2.0:1789760752.859970:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.859979:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.859982:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24489984, 24494080) 0 0 00000080:00200000:2.0:1789760752.859986:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.860003:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5979, 5979] 00020000:00200000:2.0:1789760752.860012:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24489984 chunk: [24489984, 24494080) 24494080, 4096 00020000:00200000:2.0:1789760752.860017:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.860038:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5979, 5979] 00000008:00010000:2.0:1789760752.860058:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.860064:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.860075:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24489984, 24494080) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.860091:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24489984=0x175b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.860153:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.860157:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24494080 00000020:00010000:2.0:1789760752.860166:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.860219:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.860224:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.860227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.860230:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.860238:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.860246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.860266:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760752.860392:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 24489984<>24494080 00000100:00100000:3.0:1789760752.860726:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760752.860748:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495846400:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.893089:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973296 gen 2 00000008:00010000:3.0:1789760752.893139:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760752.893147:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.893159:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.893212:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760752.893241:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495846400:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.893399:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5979, 5979] 00000080:00200000:2.0:1789760752.893418:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.893424:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.893440:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24494080 i_blocks 49159 00000020:00200000:2.0:1789760752.893444:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.893453:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.893486:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24494080, count: 4096 00000080:00200000:2.0:1789760752.893493:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24494080, bytes: 4096 00000080:00200000:2.0:1789760752.893502:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.893510:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.893513:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24494080, 24498176) 0 0 00000080:00200000:2.0:1789760752.893517:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.893543:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5980, 5980] 00020000:00200000:2.0:1789760752.893552:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24494080 chunk: [24494080, 24498176) 24498176, 4096 00020000:00200000:2.0:1789760752.893557:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.893576:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5980, 5980] 00000008:00010000:2.0:1789760752.893595:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.893600:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.893613:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24494080, 24498176) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.893626:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24494080=0x175c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.893681:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.893684:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24498176 00000020:00010000:2.0:1789760752.893693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.893697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.893701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.893705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.893708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.893716:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.893724:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.893739:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760752.893881:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 24494080<>24498176 00000100:00100000:3.0:1789760752.894028:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760752.894046:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495846528:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760752.922215:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760752.922233:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.922244:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.922267:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760752.922301:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495846528:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.922307:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5980, 5980] 00000080:00200000:2.0:1789760752.922327:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.922334:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.922350:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24498176 i_blocks 49159 00000020:00200000:2.0:1789760752.922354:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.922365:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.922400:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24498176, count: 4096 00000080:00200000:2.0:1789760752.922409:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24498176, bytes: 4096 00000080:00200000:2.0:1789760752.922419:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.922429:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.922432:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24498176, 24502272) 0 0 00000080:00200000:2.0:1789760752.922436:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.922453:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5981, 5981] 00020000:00200000:2.0:1789760752.922461:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24498176 chunk: [24498176, 24502272) 24502272, 4096 00020000:00200000:2.0:1789760752.922467:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.922487:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5981, 5981] 00000008:00010000:2.0:1789760752.922504:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.922511:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.922534:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24498176, 24502272) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.922549:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24498176=0x175d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.922609:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.922612:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24502272 00000020:00010000:2.0:1789760752.922622:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.922626:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.922630:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.922634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.922636:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.922645:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.922651:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.922667:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760752.922795:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 24498176<>24502272 00000100:00100000:3.0:1789760752.923979:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760752.924012:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495846656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760752.979219:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973298 gen 2 00000008:00010000:3.0:1789760752.979289:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760752.979298:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.979311:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.979339:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760752.979372:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495846656:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760752.979671:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5981, 5981] 00000080:00200000:2.0:1789760752.979689:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.979695:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.979712:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24502272 i_blocks 49159 00000020:00200000:2.0:1789760752.979717:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.979726:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.979762:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24502272, count: 4096 00000080:00200000:2.0:1789760752.979770:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24502272, bytes: 4096 00000080:00200000:2.0:1789760752.979781:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.979790:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.979793:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24502272, 24506368) 0 0 00000080:00200000:2.0:1789760752.979798:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.979821:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5982, 5982] 00020000:00200000:2.0:1789760752.979830:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24502272 chunk: [24502272, 24506368) 24506368, 4096 00020000:00200000:2.0:1789760752.979834:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.982265:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5982, 5982] 00000008:00010000:2.0:1789760752.982309:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.982316:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760752.982332:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24502272, 24506368) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.982350:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24502272=0x175e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.982744:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.982748:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24506368 00000020:00010000:2.0:1789760752.982758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.982762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.982766:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.982769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.982772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.982835:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.982952:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.983447:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760752.983678:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 24502272<>24506368 00000100:00100000:3.0:1789760752.984118:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760752.984147:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495846784:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760753.029249:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760753.029264:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.029275:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.029302:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760753.029329:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495846784:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760753.029425:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5982, 5982] 00000080:00200000:2.0:1789760753.029443:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.029449:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.029464:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24506368 i_blocks 49159 00000020:00200000:2.0:1789760753.029468:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.029476:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.029506:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24506368, count: 4096 00000080:00200000:2.0:1789760753.029513:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24506368, bytes: 4096 00000080:00200000:2.0:1789760753.029535:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.029544:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.029546:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24506368, 24510464) 0 0 00000080:00200000:2.0:1789760753.029550:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.029734:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5983, 5983] 00020000:00200000:2.0:1789760753.029742:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24506368 chunk: [24506368, 24510464) 24510464, 4096 00020000:00200000:2.0:1789760753.029747:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.029767:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5983, 5983] 00000008:00010000:2.0:1789760753.029783:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.029788:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760753.029800:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24506368, 24510464) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.029814:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24506368=0x175f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.029865:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.029867:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24510464 00000020:00010000:2.0:1789760753.029875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.029879:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.029883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.029886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.029889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.029896:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.029902:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.029917:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760753.030027:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 24506368<>24510464 00000100:00100000:3.0:1789760753.030084:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495846912:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760753.089585:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973300 gen 2 00000008:00010000:2.0:1789760753.089649:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760753.089658:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.089669:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.089697:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760753.089728:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495846912:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760753.089841:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5983, 5983] 00000080:00200000:0.0:1789760753.089868:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760753.089875:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760753.089890:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24510464 i_blocks 49159 00000020:00200000:0.0:1789760753.089894:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760753.089903:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760753.089938:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24510464, count: 4096 00000080:00200000:0.0:1789760753.089945:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24510464, bytes: 4096 00000080:00200000:0.0:1789760753.089956:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760753.089964:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760753.089966:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24510464, 24514560) 0 0 00000080:00200000:0.0:1789760753.089970:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760753.089987:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5984, 5984] 00020000:00200000:0.0:1789760753.089998:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24510464 chunk: [24510464, 24514560) 24514560, 4096 00020000:00200000:0.0:1789760753.090003:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760753.090022:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5984, 5984] 00000008:00010000:0.0:1789760753.090039:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760753.090045:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760753.090056:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24510464, 24514560) append 0 append lockpos 0 00000080:00200000:0.0:1789760753.090070:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24510464=0x1760000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760753.090129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760753.090132:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24514560 00000020:00010000:0.0:1789760753.090141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760753.090145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.090147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.090151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760753.090154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760753.090161:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760753.090512:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760753.090544:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760753.090676:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24510464<>24514560 00000100:00100000:0.0:1789760753.090717:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760753.090737:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495847040:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760753.099555:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:0.0:1789760753.189645:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760753.189666:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.189681:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.189781:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760753.189814:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495847040:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760753.190241:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5984, 5984] 00000080:00200000:1.0:1789760753.190262:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760753.190270:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760753.190287:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24514560 i_blocks 49159 00000020:00200000:1.0:1789760753.190291:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760753.190301:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760753.190336:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24514560, count: 4096 00000080:00200000:1.0:1789760753.197535:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24514560, bytes: 4096 00000080:00200000:1.0:1789760753.197568:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760753.197579:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760753.197582:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24514560, 24518656) 0 0 00000080:00200000:1.0:1789760753.197588:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760753.197611:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5985, 5985] 00020000:00200000:1.0:1789760753.197621:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24514560 chunk: [24514560, 24518656) 24518656, 4096 00020000:00200000:1.0:1789760753.197626:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760753.197651:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5985, 5985] 00000008:00010000:1.0:1789760753.197669:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760753.197675:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760753.197688:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24514560, 24518656) append 0 append lockpos 0 00000080:00200000:1.0:1789760753.197703:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24514560=0x1761000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760753.197775:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760753.197778:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24518656 00000020:00010000:1.0:1789760753.197787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760753.197790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.197793:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.197796:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760753.197799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760753.197811:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760753.197819:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760753.198085:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760753.198262:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 24514560<>24518656 00000100:00100000:1.0:1789760753.198310:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760753.198336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495847168:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760753.243908:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973302 gen 2 00000008:00010000:0.0:1789760753.243979:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760753.243987:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.243998:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.244071:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760753.244103:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495847168:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760753.244215:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5985, 5985] 00000080:00200000:2.0:1789760753.244233:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.244239:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.244255:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24518656 i_blocks 49159 00000020:00200000:2.0:1789760753.244258:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.244268:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.244304:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24518656, count: 4096 00000080:00200000:2.0:1789760753.244311:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24518656, bytes: 4096 00000080:00200000:2.0:1789760753.244321:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.244331:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.244334:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24518656, 24522752) 0 0 00000080:00200000:2.0:1789760753.244339:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.244357:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5986, 5986] 00020000:00200000:2.0:1789760753.244367:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24518656 chunk: [24518656, 24522752) 24522752, 4096 00020000:00200000:2.0:1789760753.244372:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.244395:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5986, 5986] 00000008:00010000:2.0:1789760753.244414:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.244421:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760753.244434:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24518656, 24522752) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.244450:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24518656=0x1762000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.244584:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.244588:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24522752 00000020:00010000:2.0:1789760753.244598:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.244603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.244606:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.244610:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.244613:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.244620:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.244628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.244645:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760753.244777:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 24518656<>24522752 00000100:00100000:3.0:1789760753.244886:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495847296:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760753.279482:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760753.279496:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.279507:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.279599:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760753.279631:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495847296:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760753.280374:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5986, 5986] 00000080:00200000:0.0:1789760753.280394:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760753.280401:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760753.280418:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24522752 i_blocks 49159 00000020:00200000:0.0:1789760753.280423:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760753.280435:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760753.280472:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24522752, count: 4096 00000080:00200000:0.0:1789760753.280479:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24522752, bytes: 4096 00000080:00200000:0.0:1789760753.280491:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760753.280500:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760753.280502:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24522752, 24526848) 0 0 00000080:00200000:0.0:1789760753.280507:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760753.280599:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5987, 5987] 00020000:00200000:0.0:1789760753.280609:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24522752 chunk: [24522752, 24526848) 24526848, 4096 00020000:00200000:0.0:1789760753.280614:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760753.280636:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5987, 5987] 00000008:00010000:0.0:1789760753.280656:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760753.280662:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760753.280675:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24522752, 24526848) append 0 append lockpos 0 00000080:00200000:0.0:1789760753.280689:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24522752=0x1763000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760753.280757:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760753.280760:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24526848 00000020:00010000:0.0:1789760753.280768:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760753.280771:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.280774:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.280778:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760753.280780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760753.280788:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760753.280795:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760753.280817:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760753.281590:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24522752<>24526848 00000100:00100000:0.0:1789760753.281642:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760753.281666:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495847424:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760753.320742:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973304 gen 2 00000008:00010000:0.0:1789760753.320810:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760753.320819:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.320829:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.320871:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760753.320905:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495847424:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760753.321070:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5987, 5987] 00000080:00200000:2.0:1789760753.321088:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.321095:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.321110:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24526848 i_blocks 49159 00000020:00200000:2.0:1789760753.321114:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.321123:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.321159:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24526848, count: 4096 00000080:00200000:2.0:1789760753.321166:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24526848, bytes: 4096 00000080:00200000:2.0:1789760753.321208:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.321217:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.321220:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24526848, 24530944) 0 0 00000080:00200000:2.0:1789760753.321225:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.321243:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5988, 5988] 00020000:00200000:2.0:1789760753.321250:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24526848 chunk: [24526848, 24530944) 24530944, 4096 00020000:00200000:2.0:1789760753.321255:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.321277:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5988, 5988] 00000008:00010000:2.0:1789760753.321295:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.321302:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760753.321314:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24526848, 24530944) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.321328:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24526848=0x1764000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.321388:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.321391:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24530944 00000020:00010000:2.0:1789760753.321399:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.321402:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.321405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.321408:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.321411:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.321418:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.321425:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.321443:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760753.321648:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 24526848<>24530944 00000100:00100000:3.0:1789760753.321725:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495847552:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760753.381884:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973305 gen 2 00000008:00010000:3.0:1789760753.381942:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760753.381950:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.381961:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.381990:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760753.382020:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495847552:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760753.385931:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5988, 5988] 00000080:00200000:0.0:1789760753.385962:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760753.385970:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760753.385989:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24530944 i_blocks 49159 00000020:00200000:0.0:1789760753.385995:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760753.386013:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760753.386049:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24530944, count: 4096 00000080:00200000:0.0:1789760753.386057:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24530944, bytes: 4096 00000080:00200000:0.0:1789760753.386073:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760753.386083:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760753.386085:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24530944, 24535040) 0 0 00000080:00200000:0.0:1789760753.386090:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760753.386106:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5989, 5989] 00020000:00200000:0.0:1789760753.386115:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24530944 chunk: [24530944, 24535040) 24535040, 4096 00020000:00200000:0.0:1789760753.386121:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760753.386143:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5989, 5989] 00000008:00010000:0.0:1789760753.386163:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760753.386242:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760753.386257:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24530944, 24535040) append 0 append lockpos 0 00000080:00200000:0.0:1789760753.386272:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24530944=0x1765000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760753.386341:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760753.386344:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24535040 00000020:00010000:0.0:1789760753.386352:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760753.386356:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.386358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.386362:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760753.386365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760753.386373:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760753.386380:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760753.386403:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760753.386662:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24530944<>24535040 00000100:00100000:0.0:1789760753.386712:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760753.386737:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495847680:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760753.447020:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973306 gen 2 00000008:00010000:0.0:1789760753.447081:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760753.447091:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.447105:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.447384:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760753.447417:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495847680:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760753.447577:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5989, 5989] 00000080:00200000:2.0:1789760753.447596:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.447603:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.447619:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24535040 i_blocks 49159 00000020:00200000:2.0:1789760753.447623:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.447634:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.447670:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24535040, count: 4096 00000080:00200000:2.0:1789760753.447678:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24535040, bytes: 4096 00000080:00200000:2.0:1789760753.447695:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.447705:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.447708:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24535040, 24539136) 0 0 00000080:00200000:2.0:1789760753.447713:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.447731:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5990, 5990] 00020000:00200000:2.0:1789760753.447740:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24535040 chunk: [24535040, 24539136) 24539136, 4096 00020000:00200000:2.0:1789760753.447745:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.447766:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5990, 5990] 00000008:00010000:2.0:1789760753.447785:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.447792:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760753.447805:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24535040, 24539136) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.447821:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24535040=0x1766000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.448050:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.448053:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24539136 00000020:00010000:2.0:1789760753.448062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.448067:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.448069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.448073:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.448076:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.448084:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.448091:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.448109:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760753.448341:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 24535040<>24539136 00000100:00100000:3.0:1789760753.448458:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495847808:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760753.504615:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973307 gen 2 00000008:00010000:3.0:1789760753.504665:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760753.504675:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.504686:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.504770:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760753.504802:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495847808:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760753.505069:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5990, 5990] 00000080:00200000:0.0:1789760753.505087:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760753.505094:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760753.505111:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24539136 i_blocks 49159 00000020:00200000:0.0:1789760753.505115:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760753.505125:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760753.505160:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24539136, count: 4096 00000080:00200000:0.0:1789760753.505212:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24539136, bytes: 4096 00000080:00200000:0.0:1789760753.505225:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760753.505234:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760753.505237:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24539136, 24543232) 0 0 00000080:00200000:0.0:1789760753.505242:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760753.505260:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5991, 5991] 00020000:00200000:0.0:1789760753.505268:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24539136 chunk: [24539136, 24543232) 24543232, 4096 00020000:00200000:0.0:1789760753.505273:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760753.505296:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5991, 5991] 00000008:00010000:0.0:1789760753.505314:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760753.505319:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760753.505331:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24539136, 24543232) append 0 append lockpos 0 00000080:00200000:0.0:1789760753.505345:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24539136=0x1767000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760753.505701:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760753.505705:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24543232 00000020:00010000:0.0:1789760753.505717:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760753.505720:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.505723:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.505726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760753.505729:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760753.505738:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760753.505747:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760753.505771:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760753.506253:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24539136<>24543232 00000100:00100000:1.0:1789760753.506756:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760753.506786:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495847936:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760753.544498:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760753.544513:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.544546:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.544567:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760753.544594:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495847936:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760753.544675:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5991, 5991] 00000080:00200000:1.0:1789760753.544692:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760753.544699:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760753.544714:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24543232 i_blocks 49159 00000020:00200000:1.0:1789760753.544717:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760753.544726:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760753.544757:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24543232, count: 4096 00000080:00200000:1.0:1789760753.544764:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24543232, bytes: 4096 00000080:00200000:1.0:1789760753.544774:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760753.544783:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760753.544786:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24543232, 24547328) 0 0 00000080:00200000:1.0:1789760753.544790:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760753.544807:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5992, 5992] 00020000:00200000:1.0:1789760753.544815:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24543232 chunk: [24543232, 24547328) 24547328, 4096 00020000:00200000:1.0:1789760753.544820:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760753.545057:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5992, 5992] 00000008:00010000:1.0:1789760753.545076:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760753.545081:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760753.545093:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24543232, 24547328) append 0 append lockpos 0 00000080:00200000:1.0:1789760753.545106:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24543232=0x1768000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760753.545197:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760753.545200:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24547328 00000020:00010000:1.0:1789760753.545208:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760753.545211:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.545215:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.545218:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760753.545221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760753.545228:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760753.545235:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760753.545423:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760753.545565:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 24543232<>24547328 00000100:00100000:0.0:1789760753.546118:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495848064:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760753.547896:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495848320:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760753.547999:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1789760753.548006:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2358:1876699495848192:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760753.548042:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495848448:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760753.555788:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2358:1876699495848192:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760753.555827:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495848448:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760753.555840:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495848320:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760753.583754:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973309 gen 2 00000008:00010000:0.0:1789760753.583838:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760753.583846:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.583857:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.583882:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760753.583913:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495848064:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760753.584215:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5992, 5992] 00000080:00200000:2.0:1789760753.584233:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.584239:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.584258:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24547328 i_blocks 49159 00000020:00200000:2.0:1789760753.584261:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.584272:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.584310:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24547328, count: 4096 00000080:00200000:2.0:1789760753.584318:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24547328, bytes: 4096 00000080:00200000:2.0:1789760753.584330:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.584337:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.584340:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24547328, 24551424) 0 0 00000080:00200000:2.0:1789760753.584345:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.584362:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5993, 5993] 00020000:00200000:2.0:1789760753.584370:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24547328 chunk: [24547328, 24551424) 24551424, 4096 00020000:00200000:2.0:1789760753.584386:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.584408:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5993, 5993] 00000008:00010000:2.0:1789760753.584426:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.584432:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760753.584444:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24547328, 24551424) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.584459:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24547328=0x1769000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.584530:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.584533:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24551424 00000020:00010000:2.0:1789760753.584542:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.584545:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.584548:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.584552:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.584555:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.584563:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.584571:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.584589:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760753.584718:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 24547328<>24551424 00000100:00100000:3.0:1789760753.584818:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495848576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760753.641537:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973310 gen 2 00000008:00010000:3.0:1789760753.641593:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760753.641601:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.641614:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.641627:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760753.641650:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495848576:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760753.641818:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5993, 5993] 00000080:00200000:0.0:1789760753.641836:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760753.641842:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760753.641901:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24551424 i_blocks 49159 00000020:00200000:0.0:1789760753.641906:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760753.641916:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760753.641953:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24551424, count: 4096 00000080:00200000:0.0:1789760753.641961:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24551424, bytes: 4096 00000080:00200000:0.0:1789760753.641972:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760753.641981:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760753.641983:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24551424, 24555520) 0 0 00000080:00200000:0.0:1789760753.641988:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760753.642004:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5994, 5994] 00020000:00200000:0.0:1789760753.642012:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24551424 chunk: [24551424, 24555520) 24555520, 4096 00020000:00200000:0.0:1789760753.642017:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760753.642039:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5994, 5994] 00000008:00010000:0.0:1789760753.642058:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760753.642063:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760753.642075:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24551424, 24555520) append 0 append lockpos 0 00000080:00200000:0.0:1789760753.642089:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24551424=0x176a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760753.642145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760753.642148:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24555520 00000020:00010000:0.0:1789760753.642163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760753.642580:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.642588:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.642592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760753.642595:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760753.642604:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760753.642611:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760753.642630:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760753.642758:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24551424<>24555520 00000100:00100000:0.0:1789760753.642798:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760753.642818:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495848704:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760753.681781:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973311 gen 2 00000008:00010000:0.0:1789760753.681832:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760753.681840:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.681851:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.681873:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760753.682027:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495848704:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760753.682358:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5994, 5994] 00000080:00200000:1.0:1789760753.682376:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760753.682382:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760753.682399:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24555520 i_blocks 49159 00000020:00200000:1.0:1789760753.682402:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760753.682412:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760753.682446:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24555520, count: 4096 00000080:00200000:1.0:1789760753.682454:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24555520, bytes: 4096 00000080:00200000:1.0:1789760753.682465:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760753.682474:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760753.682476:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24555520, 24559616) 0 0 00000080:00200000:1.0:1789760753.682481:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760753.682506:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5995, 5995] 00020000:00200000:1.0:1789760753.682525:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24555520 chunk: [24555520, 24559616) 24559616, 4096 00020000:00200000:1.0:1789760753.682533:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760753.682558:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5995, 5995] 00000008:00010000:1.0:1789760753.682576:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760753.682583:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760753.682595:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24555520, 24559616) append 0 append lockpos 0 00000080:00200000:1.0:1789760753.682609:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24555520=0x176b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760753.682666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760753.682669:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24559616 00000020:00010000:1.0:1789760753.682677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760753.682682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.682685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.682687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760753.682690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760753.682699:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760753.682706:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760753.682723:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760753.682856:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 24555520<>24559616 00000100:00100000:1.0:1789760753.683135:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760753.683154:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495848832:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760753.733389:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973312 gen 2 00000008:00010000:0.0:1789760753.733440:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760753.734701:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.734721:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.734749:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760753.734789:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495848832:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760753.734963:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5995, 5995] 00000080:00200000:1.0:1789760753.734985:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760753.734992:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760753.735009:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24559616 i_blocks 49159 00000020:00200000:1.0:1789760753.735014:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760753.735023:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760753.735063:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24559616, count: 4096 00000080:00200000:1.0:1789760753.735072:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24559616, bytes: 4096 00000080:00200000:1.0:1789760753.735087:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760753.735097:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760753.735100:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24559616, 24563712) 0 0 00000080:00200000:1.0:1789760753.735105:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760753.735129:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5996, 5996] 00020000:00200000:1.0:1789760753.735138:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24559616 chunk: [24559616, 24563712) 24563712, 4096 00020000:00200000:1.0:1789760753.735144:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760753.735738:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5996, 5996] 00000008:00010000:1.0:1789760753.735764:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760753.735770:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760753.735783:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24559616, 24563712) append 0 append lockpos 0 00000080:00200000:1.0:1789760753.735799:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24559616=0x176c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760753.735866:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760753.735869:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24563712 00000020:00010000:1.0:1789760753.735878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760753.735882:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.735885:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.735889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760753.735892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760753.737109:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760753.737122:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760753.737153:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760753.737993:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 24559616<>24563712 00000100:00100000:1.0:1789760753.738064:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760753.738090:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495848960:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760753.787711:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973313 gen 2 00000008:00010000:0.0:1789760753.787769:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760753.787778:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.787792:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.787825:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760753.787859:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495848960:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760753.788066:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5996, 5996] 00000080:00200000:1.0:1789760753.788084:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760753.788091:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760753.788106:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24563712 i_blocks 49159 00000020:00200000:1.0:1789760753.788110:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760753.788119:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760753.788153:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24563712, count: 4096 00000080:00200000:1.0:1789760753.788159:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24563712, bytes: 4096 00000080:00200000:1.0:1789760753.788205:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760753.788215:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760753.788217:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24563712, 24567808) 0 0 00000080:00200000:1.0:1789760753.788222:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760753.788238:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5997, 5997] 00020000:00200000:1.0:1789760753.788246:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24563712 chunk: [24563712, 24567808) 24567808, 4096 00020000:00200000:1.0:1789760753.788251:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760753.788273:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5997, 5997] 00000008:00010000:1.0:1789760753.788291:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760753.788297:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760753.788308:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24563712, 24567808) append 0 append lockpos 0 00000080:00200000:1.0:1789760753.788323:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24563712=0x176d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760753.788378:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760753.788380:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24567808 00000020:00010000:1.0:1789760753.788388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760753.788392:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.788395:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.788398:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760753.788401:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760753.788408:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760753.788414:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760753.788429:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760753.788570:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 24563712<>24567808 00000100:00100000:0.0:1789760753.788866:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760753.788889:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849088:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760753.833218:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973314 gen 2 00000008:00010000:0.0:1789760753.833272:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760753.833281:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.833292:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.833314:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760753.833344:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849088:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760753.833451:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5997, 5997] 00000080:00200000:1.0:1789760753.833468:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760753.833475:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760753.833490:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24567808 i_blocks 49159 00000020:00200000:1.0:1789760753.833494:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760753.833503:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760753.833547:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24567808, count: 4096 00000080:00200000:1.0:1789760753.833555:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24567808, bytes: 4096 00000080:00200000:1.0:1789760753.833566:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760753.833575:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760753.833577:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24567808, 24571904) 0 0 00000080:00200000:1.0:1789760753.833582:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760753.833599:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5998, 5998] 00020000:00200000:1.0:1789760753.833608:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24567808 chunk: [24567808, 24571904) 24571904, 4096 00020000:00200000:1.0:1789760753.833613:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760753.833633:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5998, 5998] 00000008:00010000:1.0:1789760753.833661:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760753.833668:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760753.833680:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24567808, 24571904) append 0 append lockpos 0 00000080:00200000:1.0:1789760753.833695:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24567808=0x176e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760753.833755:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760753.833758:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24571904 00000020:00010000:1.0:1789760753.833768:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760753.833772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.833775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.833779:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760753.833782:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760753.833790:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760753.833796:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760753.833813:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760753.833950:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 24567808<>24571904 00000100:00100000:0.0:1789760753.834212:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849216:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760753.926291:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973315 gen 2 00000008:00010000:0.0:1789760753.926344:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760753.926354:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760753.926365:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760753.926390:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760753.926421:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849216:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760753.926820:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5998, 5998] 00000080:00200000:1.0:1789760753.926839:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760753.926846:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760753.926868:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24571904 i_blocks 49159 00000020:00200000:1.0:1789760753.926873:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760753.926882:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760753.926919:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24571904, count: 4096 00000080:00200000:1.0:1789760753.926927:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24571904, bytes: 4096 00000080:00200000:1.0:1789760753.926940:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760753.926950:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760753.926953:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24571904, 24576000) 0 0 00000080:00200000:1.0:1789760753.926958:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760753.926985:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [5999, 5999] 00020000:00200000:1.0:1789760753.926994:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24571904 chunk: [24571904, 24576000) 24576000, 4096 00020000:00200000:1.0:1789760753.927000:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760753.927022:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5999, 5999] 00000008:00010000:1.0:1789760753.927043:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760753.927049:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760753.927063:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24571904, 24576000) append 0 append lockpos 0 00000080:00200000:1.0:1789760753.927079:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24571904=0x176f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760753.927145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760753.927148:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24576000 00000020:00010000:1.0:1789760753.927157:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760753.927161:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.927165:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760753.927219:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760753.927223:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760753.927232:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760753.927240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760753.927258:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760753.927391:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 24571904<>24576000 00000100:00100000:0.0:1789760753.928744:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760753.928779:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849344:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760754.003111:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973316 gen 2 00000008:00010000:0.0:1789760754.003162:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760754.003224:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.003236:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.003261:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760754.003282:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849344:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760754.004926:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [5999, 5999] 00000080:00200000:1.0:1789760754.004952:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.004959:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.004988:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24576000 i_blocks 49159 00000020:00200000:1.0:1789760754.004993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.005005:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.005041:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24576000, count: 4096 00000080:00200000:1.0:1789760754.005049:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24576000, bytes: 4096 00000080:00200000:1.0:1789760754.005064:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.005074:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.005077:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24576000, 24580096) 0 0 00000080:00200000:1.0:1789760754.005082:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.005099:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6000, 6000] 00020000:00200000:1.0:1789760754.005107:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24576000 chunk: [24576000, 24580096) 24580096, 4096 00020000:00200000:1.0:1789760754.005112:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.005133:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6000, 6000] 00000008:00010000:1.0:1789760754.005150:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.005156:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760754.005208:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24576000, 24580096) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.005226:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24576000=0x1770000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.005288:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.005291:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24580096 00000020:00010000:1.0:1789760754.005299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.005304:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.005307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.005311:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.005314:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.005322:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.005329:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.005345:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760754.005726:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 24576000<>24580096 00000100:00100000:0.0:1789760754.006363:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760754.006392:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495849472:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760754.041785:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973317 gen 2 00000008:00010000:0.0:1789760754.041844:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760754.041853:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.041864:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.041887:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760754.041918:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495849472:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760754.041973:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6000, 6000] 00000080:00200000:1.0:1789760754.041990:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.041997:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.042013:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24580096 i_blocks 49159 00000020:00200000:1.0:1789760754.042016:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.042026:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.042058:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24580096, count: 4096 00000080:00200000:1.0:1789760754.042065:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24580096, bytes: 4096 00000080:00200000:1.0:1789760754.042076:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.042085:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.042087:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24580096, 24584192) 0 0 00000080:00200000:1.0:1789760754.042092:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.042108:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6001, 6001] 00020000:00200000:1.0:1789760754.042117:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24580096 chunk: [24580096, 24584192) 24584192, 4096 00020000:00200000:1.0:1789760754.042122:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.042142:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6001, 6001] 00000008:00010000:1.0:1789760754.042161:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.042194:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760754.042207:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24580096, 24584192) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.042224:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24580096=0x1771000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.042281:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.042284:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24584192 00000020:00010000:1.0:1789760754.042293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.042297:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.042301:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.042304:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.042308:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.042315:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.042322:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.042338:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760754.042465:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 24580096<>24584192 00000100:00100000:0.0:1789760754.043481:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760754.043508:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849600:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760754.080477:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973318 gen 2 00000008:00010000:0.0:1789760754.080603:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760754.080612:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.080622:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.080691:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760754.080723:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849600:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760754.080928:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6001, 6001] 00000080:00200000:1.0:1789760754.080945:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.080951:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.080967:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24584192 i_blocks 49159 00000020:00200000:1.0:1789760754.080971:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.080980:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.081013:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24584192, count: 4096 00000080:00200000:1.0:1789760754.081020:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24584192, bytes: 4096 00000080:00200000:1.0:1789760754.081030:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.081039:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.081042:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24584192, 24588288) 0 0 00000080:00200000:1.0:1789760754.081046:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.081063:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6002, 6002] 00020000:00200000:1.0:1789760754.081071:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24584192 chunk: [24584192, 24588288) 24588288, 4096 00020000:00200000:1.0:1789760754.081075:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.081095:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6002, 6002] 00000008:00010000:1.0:1789760754.081114:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.081121:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760754.081133:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24584192, 24588288) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.081148:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24584192=0x1772000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.081809:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.081814:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24588288 00000020:00010000:1.0:1789760754.081825:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.081829:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.081832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.081836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.081839:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.081847:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.081855:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.081875:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760754.081991:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 24584192<>24588288 00000100:00100000:0.0:1789760754.082053:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760754.082075:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495849728:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760754.114044:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973319 gen 2 00000008:00010000:0.0:1789760754.114095:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760754.114104:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.114116:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.114138:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760754.114209:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495849728:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760754.114319:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6002, 6002] 00000080:00200000:1.0:1789760754.114336:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.114342:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.114359:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24588288 i_blocks 49159 00000020:00200000:1.0:1789760754.114363:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.114372:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.114407:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24588288, count: 4096 00000080:00200000:1.0:1789760754.114414:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24588288, bytes: 4096 00000080:00200000:1.0:1789760754.114424:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.114433:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.114435:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24588288, 24592384) 0 0 00000080:00200000:1.0:1789760754.114439:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.114457:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6003, 6003] 00020000:00200000:1.0:1789760754.114466:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24588288 chunk: [24588288, 24592384) 24592384, 4096 00020000:00200000:1.0:1789760754.114471:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.114490:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6003, 6003] 00000008:00010000:1.0:1789760754.114532:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.114539:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760754.114551:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24588288, 24592384) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.114566:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24588288=0x1773000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.114617:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.114620:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24592384 00000020:00010000:1.0:1789760754.114627:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.114632:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.114635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.114638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.114641:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.114648:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.114655:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.114670:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760754.114788:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 24588288<>24592384 00000100:00100000:1.0:1789760754.114828:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760754.115096:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849856:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760754.124733:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760754.147831:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973320 gen 2 00000008:00010000:0.0:1789760754.147881:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760754.147890:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.147901:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.147927:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760754.147957:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495849856:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760754.148011:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6003, 6003] 00000080:00200000:2.0:1789760754.148029:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.148036:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.148052:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24592384 i_blocks 49159 00000020:00200000:2.0:1789760754.148056:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.148067:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.148101:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24592384, count: 4096 00000080:00200000:2.0:1789760754.148109:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24592384, bytes: 4096 00000080:00200000:2.0:1789760754.148121:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.148130:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.148133:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24592384, 24596480) 0 0 00000080:00200000:2.0:1789760754.148138:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.148155:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6004, 6004] 00020000:00200000:2.0:1789760754.148164:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24592384 chunk: [24592384, 24596480) 24596480, 4096 00020000:00200000:2.0:1789760754.148206:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.148245:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6004, 6004] 00000008:00010000:2.0:1789760754.148264:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.148270:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.148283:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24592384, 24596480) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.148297:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24592384=0x1774000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.148357:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.148360:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24596480 00000020:00010000:2.0:1789760754.148369:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.148372:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.148376:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.148379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.148382:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.148388:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.148395:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.148407:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760754.148536:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 24592384<>24596480 00000100:00100000:3.0:1789760754.148636:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495849984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760754.189407:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973321 gen 2 00000008:00010000:3.0:1789760754.190770:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760754.190783:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.190800:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.190830:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000080:00200000:2.0:1789760754.191311:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6004, 6004] 00000080:00200000:2.0:1789760754.191332:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.191340:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.191359:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24596480 i_blocks 49159 00000020:00200000:2.0:1789760754.191364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.191375:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.191409:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24596480, count: 4096 00000080:00200000:2.0:1789760754.191419:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24596480, bytes: 4096 00000080:00200000:2.0:1789760754.191433:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.191442:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.191444:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24596480, 24600576) 0 0 00000080:00200000:2.0:1789760754.191448:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.191465:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6005, 6005] 00020000:00200000:2.0:1789760754.191474:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24596480 chunk: [24596480, 24600576) 24600576, 4096 00020000:00200000:2.0:1789760754.191479:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1789760754.191751:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495849984:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760754.192077:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6005, 6005] 00000008:00010000:2.0:1789760754.192099:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.192105:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.192117:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24596480, 24600576) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.192133:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24596480=0x1775000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.192213:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.192216:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24600576 00000020:00010000:2.0:1789760754.192224:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.192228:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.192231:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.192235:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.192238:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.192245:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.192253:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.192271:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760754.192398:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 24596480<>24600576 00000100:00100000:3.0:1789760754.192675:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760754.192693:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495850112:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760754.228135:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973322 gen 2 00000008:00010000:2.0:1789760754.228772:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760754.228784:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.228995:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.229096:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760754.229130:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495850112:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760754.229363:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6005, 6005] 00000080:00200000:0.0:1789760754.229381:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760754.229387:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760754.229404:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24600576 i_blocks 49159 00000020:00200000:0.0:1789760754.229408:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760754.229418:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760754.229727:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24600576, count: 4096 00000080:00200000:0.0:1789760754.229737:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24600576, bytes: 4096 00000080:00200000:0.0:1789760754.229761:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760754.229772:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760754.229774:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24600576, 24604672) 0 0 00000080:00200000:0.0:1789760754.229779:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760754.229983:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6006, 6006] 00020000:00200000:0.0:1789760754.229993:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24600576 chunk: [24600576, 24604672) 24604672, 4096 00020000:00200000:0.0:1789760754.229998:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760754.230022:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6006, 6006] 00000008:00010000:0.0:1789760754.230041:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760754.230047:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760754.230060:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24600576, 24604672) append 0 append lockpos 0 00000080:00200000:0.0:1789760754.230461:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24600576=0x1776000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760754.230544:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760754.230547:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24604672 00000020:00010000:0.0:1789760754.230556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760754.230560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.230563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.230566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760754.230569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760754.230579:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760754.230588:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760754.230617:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760754.230773:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 24600576<>24604672 00000100:00100000:0.0:1789760754.231025:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760754.231052:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495850240:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760754.257159:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973323 gen 2 00000008:00010000:1.0:1789760754.257234:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760754.257245:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760754.257259:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760754.257291:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760754.257328:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495850240:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760754.257800:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6006, 6006] 00000080:00200000:0.0:1789760754.257972:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760754.257980:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760754.257996:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24604672 i_blocks 49159 00000020:00200000:0.0:1789760754.258001:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760754.258010:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760754.258050:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24604672, count: 4096 00000080:00200000:0.0:1789760754.258058:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24604672, bytes: 4096 00000080:00200000:0.0:1789760754.258072:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760754.258081:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760754.258084:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24604672, 24608768) 0 0 00000080:00200000:0.0:1789760754.258089:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760754.258108:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6007, 6007] 00020000:00200000:0.0:1789760754.258118:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24604672 chunk: [24604672, 24608768) 24608768, 4096 00020000:00200000:0.0:1789760754.258123:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760754.258145:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6007, 6007] 00000008:00010000:0.0:1789760754.258166:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760754.258301:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760754.258314:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24604672, 24608768) append 0 append lockpos 0 00000080:00200000:0.0:1789760754.258329:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24604672=0x1777000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760754.258398:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760754.258402:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24608768 00000020:00010000:0.0:1789760754.258410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760754.258413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.258416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.258420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760754.258422:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760754.258431:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760754.258438:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760754.258457:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760754.258616:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 24604672<>24608768 00000100:00100000:1.0:1789760754.260301:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760754.260340:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495850368:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760754.318287:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760754.318306:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.318317:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.318343:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760754.318373:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495850368:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760754.318496:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6007, 6007] 00000080:00200000:2.0:1789760754.318525:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.318532:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.318746:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24608768 i_blocks 49159 00000020:00200000:2.0:1789760754.318751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.318762:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.318800:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24608768, count: 4096 00000080:00200000:2.0:1789760754.318808:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24608768, bytes: 4096 00000080:00200000:2.0:1789760754.318819:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.318829:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.318832:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24608768, 24612864) 0 0 00000080:00200000:2.0:1789760754.318837:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.318854:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6008, 6008] 00020000:00200000:2.0:1789760754.318864:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24608768 chunk: [24608768, 24612864) 24612864, 4096 00020000:00200000:2.0:1789760754.318869:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.318890:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6008, 6008] 00000008:00010000:2.0:1789760754.318911:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.318918:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.318931:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24608768, 24612864) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.318946:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24608768=0x1778000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.319104:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.319108:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24612864 00000020:00010000:2.0:1789760754.319118:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.319121:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.319125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.319129:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.319132:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.319140:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.319148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.319219:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760754.319352:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 24608768<>24612864 00000100:00100000:3.0:1789760754.319533:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495850496:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760754.380407:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973325 gen 2 00000008:00010000:3.0:1789760754.380602:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760754.380611:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.380623:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.380650:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760754.380685:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495850496:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760754.381214:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6008, 6008] 00000080:00200000:2.0:1789760754.381234:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.381240:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.381257:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24612864 i_blocks 49159 00000020:00200000:2.0:1789760754.381261:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.381271:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.381309:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24612864, count: 4096 00000080:00200000:2.0:1789760754.381317:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24612864, bytes: 4096 00000080:00200000:2.0:1789760754.381330:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.381342:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.381345:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24612864, 24616960) 0 0 00000080:00200000:2.0:1789760754.381349:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.381365:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6009, 6009] 00020000:00200000:2.0:1789760754.381374:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24612864 chunk: [24612864, 24616960) 24616960, 4096 00020000:00200000:2.0:1789760754.381379:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.381400:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6009, 6009] 00000008:00010000:2.0:1789760754.381419:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.381425:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.381436:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24612864, 24616960) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.381450:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24612864=0x1779000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.381509:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.381523:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24616960 00000020:00010000:2.0:1789760754.381531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.381534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.381537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.381540:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.381543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.381550:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.381558:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.381577:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760754.381705:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 24612864<>24616960 00000100:00100000:3.0:1789760754.381945:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760754.381966:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495850624:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760754.465479:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973326 gen 2 00000008:00010000:3.0:1789760754.465554:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760754.465562:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.465574:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.465598:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760754.465631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495850624:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760754.465643:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6009, 6009] 00000080:00200000:2.0:1789760754.465662:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.465669:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.465687:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24616960 i_blocks 49159 00000020:00200000:2.0:1789760754.465692:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.465704:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.465740:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24616960, count: 4096 00000080:00200000:2.0:1789760754.465748:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24616960, bytes: 4096 00000080:00200000:2.0:1789760754.465757:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.465767:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.465770:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24616960, 24621056) 0 0 00000080:00200000:2.0:1789760754.465774:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.465791:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6010, 6010] 00020000:00200000:2.0:1789760754.465800:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24616960 chunk: [24616960, 24621056) 24621056, 4096 00020000:00200000:2.0:1789760754.465805:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.465825:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6010, 6010] 00000008:00010000:2.0:1789760754.465844:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.465850:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.465861:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24616960, 24621056) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.465877:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24616960=0x177a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.465936:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.465939:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24621056 00000020:00010000:2.0:1789760754.465948:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.465952:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.466625:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.466633:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.466636:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.466646:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.466654:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.466683:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760754.466827:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 24616960<>24621056 00000100:00100000:3.0:1789760754.466912:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495850752:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760754.567113:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973327 gen 2 00000008:00010000:3.0:1789760754.567165:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760754.567207:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.567220:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.567249:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760754.567282:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495850752:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760754.567471:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6010, 6010] 00000080:00200000:0.0:1789760754.567488:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760754.567495:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760754.567524:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24621056 i_blocks 49159 00000020:00200000:0.0:1789760754.567528:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760754.567539:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760754.567650:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24621056, count: 4096 00000080:00200000:0.0:1789760754.567659:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24621056, bytes: 4096 00000080:00200000:0.0:1789760754.567671:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760754.567681:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760754.567684:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24621056, 24625152) 0 0 00000080:00200000:0.0:1789760754.567688:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760754.567708:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6011, 6011] 00020000:00200000:0.0:1789760754.567718:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24621056 chunk: [24621056, 24625152) 24625152, 4096 00020000:00200000:0.0:1789760754.567724:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760754.567746:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6011, 6011] 00000008:00010000:0.0:1789760754.567765:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760754.567771:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760754.567783:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24621056, 24625152) append 0 append lockpos 0 00000080:00200000:0.0:1789760754.567797:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24621056=0x177b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760754.567856:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760754.567859:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24625152 00000020:00010000:0.0:1789760754.567868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760754.567872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.567875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.567878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760754.567881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760754.567888:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760754.567897:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760754.567912:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760754.568040:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 24621056<>24625152 00000100:00100000:1.0:1789760754.568116:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495850880:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760754.620332:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973328 gen 2 00000008:00010000:0.0:1789760754.620385:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760754.620395:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.620407:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.620440:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760754.620471:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495850880:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760754.620687:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6011, 6011] 00000080:00200000:2.0:1789760754.620703:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.620710:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.620727:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24625152 i_blocks 49159 00000020:00200000:2.0:1789760754.620732:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.620741:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.620776:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24625152, count: 4096 00000080:00200000:2.0:1789760754.620783:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24625152, bytes: 4096 00000080:00200000:2.0:1789760754.620793:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.620804:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.620807:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24625152, 24629248) 0 0 00000080:00200000:2.0:1789760754.620812:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.620829:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6012, 6012] 00020000:00200000:2.0:1789760754.620838:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24625152 chunk: [24625152, 24629248) 24629248, 4096 00020000:00200000:2.0:1789760754.620843:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.620862:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6012, 6012] 00000008:00010000:2.0:1789760754.620882:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.620890:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.620903:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24625152, 24629248) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.620918:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24625152=0x177c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.621205:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.621208:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24629248 00000020:00010000:2.0:1789760754.621218:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.621222:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.621225:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd218376ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.621229:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.621232:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.621239:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.621246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.621261:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760754.621379:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 24625152<>24629248 00000100:00100000:3.0:1789760754.621505:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495851008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760754.677959:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973329 gen 2 00000008:00010000:3.0:1789760754.679124:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760754.679137:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.679150:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.679255:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760754.679286:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495851008:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760754.680362:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6012, 6012] 00000080:00200000:0.0:1789760754.680395:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760754.680402:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.680501:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24629248 i_blocks 49159 00000020:00200000:2.0:1789760754.680521:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.680533:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.680567:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24629248, count: 4096 00000080:00200000:2.0:1789760754.680575:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24629248, bytes: 4096 00000080:00200000:2.0:1789760754.680589:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.680597:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.680599:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24629248, 24633344) 0 0 00000080:00200000:2.0:1789760754.680604:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.680621:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6013, 6013] 00020000:00200000:2.0:1789760754.680629:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24629248 chunk: [24629248, 24633344) 24633344, 4096 00020000:00200000:2.0:1789760754.680633:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.680655:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6013, 6013] 00000008:00010000:2.0:1789760754.680673:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.680679:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.680690:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24629248, 24633344) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.680703:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24629248=0x177d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.680765:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.680767:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24633344 00000020:00010000:2.0:1789760754.680775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.680778:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.680781:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.680784:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.680786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.680793:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.680800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.680846:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760754.680985:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 24629248<>24633344 00000100:00100000:3.0:1789760754.681389:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495851136:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760754.735919:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973330 gen 2 00000008:00010000:3.0:1789760754.735974:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760754.735983:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.735995:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.736023:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760754.736054:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495851136:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760754.736225:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6013, 6013] 00000080:00200000:2.0:1789760754.736243:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.736250:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.736268:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24633344 i_blocks 49159 00000020:00200000:2.0:1789760754.736272:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.736283:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.736319:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24633344, count: 4096 00000080:00200000:2.0:1789760754.736327:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24633344, bytes: 4096 00000080:00200000:2.0:1789760754.736338:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.736347:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.736350:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24633344, 24637440) 0 0 00000080:00200000:2.0:1789760754.736355:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.736371:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6014, 6014] 00020000:00200000:2.0:1789760754.736379:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24633344 chunk: [24633344, 24637440) 24637440, 4096 00020000:00200000:2.0:1789760754.736384:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.736403:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6014, 6014] 00000008:00010000:2.0:1789760754.736420:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.736426:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.736438:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24633344, 24637440) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.736453:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24633344=0x177e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.736556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.736560:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24637440 00000020:00010000:2.0:1789760754.736569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.736573:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.736576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.736579:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.736581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.736588:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.736595:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.736612:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760754.736815:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 24633344<>24637440 00000100:00100000:3.0:1789760754.736892:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495851264:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760754.782244:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973331 gen 2 00000008:00010000:3.0:1789760754.782301:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760754.782311:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.782323:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.782348:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760754.782379:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495851264:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760754.782491:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6014, 6014] 00000080:00200000:1.0:1789760754.782518:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.782525:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.782541:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24637440 i_blocks 49159 00000020:00200000:1.0:1789760754.782544:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.782555:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.782586:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24637440, count: 4096 00000080:00200000:1.0:1789760754.782593:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24637440, bytes: 4096 00000080:00200000:1.0:1789760754.782604:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.782614:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.782616:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24637440, 24641536) 0 0 00000080:00200000:1.0:1789760754.782621:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.782638:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6015, 6015] 00020000:00200000:1.0:1789760754.782647:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24637440 chunk: [24637440, 24641536) 24641536, 4096 00020000:00200000:1.0:1789760754.782652:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.782672:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6015, 6015] 00000008:00010000:1.0:1789760754.782691:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.782697:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760754.782710:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24637440, 24641536) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.782723:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24637440=0x177f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.782779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.782782:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24641536 00000020:00010000:1.0:1789760754.782790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.782794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.782797:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.782801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.782805:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.782812:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.782819:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.782847:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760754.782979:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 24637440<>24641536 00000100:00100000:0.0:1789760754.785011:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495851392:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760754.849923:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973332 gen 2 00000008:00010000:0.0:1789760754.849974:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760754.849984:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.850006:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.850031:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760754.850063:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495851392:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760754.850571:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6015, 6015] 00000080:00200000:1.0:1789760754.850589:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.850596:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.850617:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24641536 i_blocks 49159 00000020:00200000:1.0:1789760754.850621:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.850630:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.850664:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24641536, count: 4096 00000080:00200000:1.0:1789760754.850671:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24641536, bytes: 4096 00000080:00200000:1.0:1789760754.850683:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.850692:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.850694:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24641536, 24645632) 0 0 00000080:00200000:1.0:1789760754.850698:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.850714:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6016, 6016] 00020000:00200000:1.0:1789760754.850723:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24641536 chunk: [24641536, 24645632) 24645632, 4096 00020000:00200000:1.0:1789760754.850728:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.850747:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6016, 6016] 00000008:00010000:1.0:1789760754.850765:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.850771:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760754.850783:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24641536, 24645632) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.850797:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24641536=0x1780000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.850854:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.850857:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24645632 00000020:00010000:1.0:1789760754.850866:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.850870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.850873:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.850877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.850880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.850888:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.850895:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.850911:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760754.851050:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 24641536<>24645632 00000100:00100000:0.0:1789760754.851389:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495851520:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760754.902464:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973333 gen 2 00000008:00010000:0.0:1789760754.902581:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760754.902591:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.902603:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.902631:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760754.902663:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495851520:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760754.902790:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6016, 6016] 00000080:00200000:2.0:1789760754.902807:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.902814:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.902829:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24645632 i_blocks 49159 00000020:00200000:2.0:1789760754.902833:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.902843:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.902877:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24645632, count: 4096 00000080:00200000:2.0:1789760754.902884:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24645632, bytes: 4096 00000080:00200000:2.0:1789760754.902896:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.902904:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.902906:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24645632, 24649728) 0 0 00000080:00200000:2.0:1789760754.902911:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.902929:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6017, 6017] 00020000:00200000:2.0:1789760754.902939:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24645632 chunk: [24645632, 24649728) 24649728, 4096 00020000:00200000:2.0:1789760754.902944:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.902964:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6017, 6017] 00000008:00010000:2.0:1789760754.902981:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.902986:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760754.903000:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24645632, 24649728) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.903015:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24645632=0x1781000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.903074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.903077:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24649728 00000020:00010000:2.0:1789760754.903086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.903090:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.903093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.903096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.903099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.903106:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.903112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.903129:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760754.903362:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 24645632<>24649728 00000100:00100000:2.0:1789760754.903411:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495851648:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760754.944213:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973334 gen 2 00000008:00010000:2.0:1789760754.944269:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760754.944278:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.944289:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.944556:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760754.944588:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495851648:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760754.944758:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6017, 6017] 00000080:00200000:3.0:1789760754.944786:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.944793:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.944810:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24649728 i_blocks 49159 00000020:00200000:3.0:1789760754.944814:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.944824:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.944862:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24649728, count: 4096 00000080:00200000:3.0:1789760754.944870:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24649728, bytes: 4096 00000080:00200000:3.0:1789760754.944880:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.944890:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.944893:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24649728, 24653824) 0 0 00000080:00200000:3.0:1789760754.944897:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.944914:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6018, 6018] 00020000:00200000:3.0:1789760754.944921:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24649728 chunk: [24649728, 24653824) 24653824, 4096 00020000:00200000:3.0:1789760754.944926:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.944946:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6018, 6018] 00000008:00010000:3.0:1789760754.944963:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.944969:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760754.944981:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24649728, 24653824) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.944997:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24649728=0x1782000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.945517:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.945521:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24653824 00000020:00010000:3.0:1789760754.945531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.945535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.945538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.945542:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.945544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.945552:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.945559:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.945580:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760754.945762:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 24649728<>24653824 00000100:00100000:2.0:1789760754.945897:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495851776:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760755.013551:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973335 gen 2 00000008:00010000:2.0:1789760755.015143:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760755.015426:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.015440:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.015473:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760755.015582:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495851776:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760755.015589:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6018, 6018] 00000080:00200000:3.0:1789760755.015611:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.015617:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.015636:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24653824 i_blocks 49159 00000020:00200000:3.0:1789760755.015641:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.015652:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.015691:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24653824, count: 4096 00000080:00200000:3.0:1789760755.015701:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24653824, bytes: 4096 00000080:00200000:3.0:1789760755.015718:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.015727:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.015729:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24653824, 24657920) 0 0 00000080:00200000:3.0:1789760755.015733:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.015751:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6019, 6019] 00020000:00200000:3.0:1789760755.015759:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24653824 chunk: [24653824, 24657920) 24657920, 4096 00020000:00200000:3.0:1789760755.015763:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.015785:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6019, 6019] 00000008:00010000:3.0:1789760755.015800:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.015805:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760755.015815:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24653824, 24657920) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.015842:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24653824=0x1783000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.015909:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.015912:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24657920 00000020:00010000:3.0:1789760755.015921:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.015924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.015927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.015930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.015933:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.015941:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.015948:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.015965:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760755.016156:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 24653824<>24657920 00000100:00100000:2.0:1789760755.016578:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495851904:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760755.053137:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760755.053151:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.053163:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.053262:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:2.0:1789760755.053290:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495851904:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760755.053391:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6019, 6019] 00000080:00200000:3.0:1789760755.053408:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.053414:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.053429:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24657920 i_blocks 49159 00000020:00200000:3.0:1789760755.053434:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.053443:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.053477:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24657920, count: 4096 00000080:00200000:3.0:1789760755.053559:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24657920, bytes: 4096 00000080:00200000:3.0:1789760755.053657:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.053667:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.053670:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24657920, 24662016) 0 0 00000080:00200000:3.0:1789760755.053675:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.053695:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6020, 6020] 00020000:00200000:3.0:1789760755.053703:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24657920 chunk: [24657920, 24662016) 24662016, 4096 00020000:00200000:3.0:1789760755.053709:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.053729:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6020, 6020] 00000008:00010000:3.0:1789760755.053747:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.053753:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760755.053764:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24657920, 24662016) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.053778:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24657920=0x1784000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.053967:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.053971:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24662016 00000020:00010000:3.0:1789760755.053980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.053984:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.053988:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.053991:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.053994:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.054003:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.054010:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.054030:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760755.054155:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 24657920<>24662016 00000100:00100000:2.0:1789760755.054262:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495852032:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760755.119771:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973337 gen 2 00000008:00010000:2.0:1789760755.119845:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:2.0:1789760755.119859:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.119869:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.119889:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000080:00200000:1.0:1789760755.120028:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6020, 6020] 00000080:00200000:1.0:1789760755.120046:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760755.120053:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760755.120070:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24662016 i_blocks 49159 00000020:00200000:1.0:1789760755.120074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760755.120083:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760755.120116:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24662016, count: 4096 00000080:00200000:1.0:1789760755.120124:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24662016, bytes: 4096 00000080:00200000:1.0:1789760755.120137:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760755.120147:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760755.120149:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24662016, 24666112) 0 0 00000080:00200000:1.0:1789760755.120154:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760755.120200:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6021, 6021] 00020000:00200000:1.0:1789760755.120208:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24662016 chunk: [24662016, 24666112) 24666112, 4096 00020000:00200000:1.0:1789760755.120212:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760755.120232:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6021, 6021] 00000008:00010000:1.0:1789760755.120249:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760755.120254:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760755.120265:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24662016, 24666112) append 0 append lockpos 0 00000080:00200000:1.0:1789760755.120279:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24662016=0x1785000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760755.120336:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760755.120338:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24666112 00000020:00010000:1.0:1789760755.120346:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760755.120350:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.120352:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.120356:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760755.120359:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760755.120365:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760755.120371:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760755.120389:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760755.120521:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24662016<>24666112 00000100:00100000:0.0:1789760755.120646:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852160:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760755.121036:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495852032:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760755.147709:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760755.199990:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973338 gen 2 00000008:00010000:0.0:1789760755.200028:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760755.200036:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760755.200045:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760755.200212:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760755.200232:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852160:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760755.201388:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6021, 6021] 00000080:00200000:1.0:1789760755.201411:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760755.201419:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760755.201793:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24666112 i_blocks 49159 00000020:00200000:1.0:1789760755.201800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760755.201818:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760755.201849:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24666112, count: 4096 00000080:00200000:1.0:1789760755.201856:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24666112, bytes: 4096 00000080:00200000:1.0:1789760755.201869:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760755.201877:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760755.201880:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24666112, 24670208) 0 0 00000080:00200000:1.0:1789760755.201885:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760755.201901:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6022, 6022] 00020000:00200000:1.0:1789760755.201910:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24666112 chunk: [24666112, 24670208) 24670208, 4096 00020000:00200000:1.0:1789760755.201915:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760755.201932:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6022, 6022] 00000008:00010000:1.0:1789760755.201947:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760755.201953:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760755.201965:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24666112, 24670208) append 0 append lockpos 0 00000080:00200000:1.0:1789760755.201978:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24666112=0x1786000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760755.202044:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760755.202049:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24670208 00000020:00010000:1.0:1789760755.202057:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760755.202061:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.202465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.202471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760755.202474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760755.202485:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760755.202496:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760755.202535:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760755.202822:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 24666112<>24670208 00000100:00100000:0.0:1789760755.202981:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760755.203013:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852288:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760755.275991:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973339 gen 2 00000008:00010000:0.0:1789760755.276027:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760755.276034:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760755.276043:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760755.276065:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760755.276090:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852288:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760755.276196:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6022, 6022] 00000080:00200000:1.0:1789760755.276208:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760755.276214:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760755.276226:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24670208 i_blocks 49159 00000020:00200000:1.0:1789760755.276229:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760755.276250:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760755.276275:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24670208, count: 4096 00000080:00200000:1.0:1789760755.276281:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24670208, bytes: 4096 00000080:00200000:1.0:1789760755.276289:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760755.276296:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760755.276298:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24670208, 24674304) 0 0 00000080:00200000:1.0:1789760755.276302:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760755.276313:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6023, 6023] 00020000:00200000:1.0:1789760755.276321:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24670208 chunk: [24670208, 24674304) 24674304, 4096 00020000:00200000:1.0:1789760755.276325:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760755.276340:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6023, 6023] 00000008:00010000:1.0:1789760755.276362:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760755.276367:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760755.276376:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24670208, 24674304) append 0 append lockpos 0 00000080:00200000:1.0:1789760755.276386:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24670208=0x1787000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760755.276429:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760755.276432:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24674304 00000020:00010000:1.0:1789760755.276439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760755.276442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.276445:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.276448:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760755.276450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760755.276456:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760755.276461:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760755.276475:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760755.276582:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 24670208<>24674304 00000100:00100000:0.0:1789760755.276947:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852416:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760755.357910:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973340 gen 2 00000008:00010000:0.0:1789760755.357944:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760755.357951:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760755.357959:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760755.357979:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760755.358003:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852416:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760755.358144:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6023, 6023] 00000080:00200000:1.0:1789760755.358156:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760755.358161:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760755.358195:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24674304 i_blocks 49159 00000020:00200000:1.0:1789760755.358199:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760755.358207:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760755.358245:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24674304, count: 4096 00000080:00200000:1.0:1789760755.358251:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24674304, bytes: 4096 00000080:00200000:1.0:1789760755.358259:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760755.358266:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760755.358269:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24674304, 24678400) 0 0 00000080:00200000:1.0:1789760755.358273:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760755.358286:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6024, 6024] 00020000:00200000:1.0:1789760755.358293:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24674304 chunk: [24674304, 24678400) 24678400, 4096 00020000:00200000:1.0:1789760755.358297:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760755.358312:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6024, 6024] 00000008:00010000:1.0:1789760755.358324:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760755.358329:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760755.358338:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24674304, 24678400) append 0 append lockpos 0 00000080:00200000:1.0:1789760755.358349:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24674304=0x1788000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760755.358392:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760755.358395:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24678400 00000020:00010000:1.0:1789760755.358401:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760755.358404:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.358407:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.358410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760755.358412:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760755.358418:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760755.358424:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760755.358437:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760755.358540:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 24674304<>24678400 00000100:00100000:0.0:1789760755.358672:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760755.358685:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852544:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760755.415066:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973341 gen 2 00000008:00010000:0.0:1789760755.415203:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760755.415333:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760755.415344:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760755.415656:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760755.415682:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852544:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760755.420614:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6024, 6024] 00000080:00200000:1.0:1789760755.420638:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760755.420644:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760755.420657:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24678400 i_blocks 49159 00000020:00200000:1.0:1789760755.420662:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760755.420675:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760755.420700:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24678400, count: 4096 00000080:00200000:1.0:1789760755.420706:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24678400, bytes: 4096 00000080:00200000:1.0:1789760755.420719:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760755.420726:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760755.420728:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24678400, 24682496) 0 0 00000080:00200000:1.0:1789760755.420732:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760755.420746:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6025, 6025] 00020000:00200000:1.0:1789760755.420752:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24678400 chunk: [24678400, 24682496) 24682496, 4096 00020000:00200000:1.0:1789760755.420756:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760755.420775:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6025, 6025] 00000008:00010000:1.0:1789760755.420790:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760755.420795:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760755.420804:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24678400, 24682496) append 0 append lockpos 0 00000080:00200000:1.0:1789760755.420825:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24678400=0x1789000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760755.421154:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760755.421157:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24682496 00000020:00010000:1.0:1789760755.421164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760755.421270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.421276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.421279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760755.421282:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760755.421289:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760755.421295:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760755.421319:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760755.421515:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 24678400<>24682496 00000100:00100000:0.0:1789760755.421584:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852672:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760755.504310:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973342 gen 2 00000008:00010000:0.0:1789760755.504348:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760755.504354:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760755.504363:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760755.504424:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760755.504453:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852672:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760755.504604:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6025, 6025] 00000080:00200000:1.0:1789760755.504617:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760755.504621:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760755.504635:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24682496 i_blocks 49159 00000020:00200000:1.0:1789760755.504638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760755.504645:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760755.504671:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24682496, count: 4096 00000080:00200000:1.0:1789760755.504676:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24682496, bytes: 4096 00000080:00200000:1.0:1789760755.504684:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760755.504692:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760755.504694:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24682496, 24686592) 0 0 00000080:00200000:1.0:1789760755.504698:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760755.504710:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6026, 6026] 00020000:00200000:1.0:1789760755.504718:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24682496 chunk: [24682496, 24686592) 24686592, 4096 00020000:00200000:1.0:1789760755.504721:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760755.504736:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6026, 6026] 00000008:00010000:1.0:1789760755.504748:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760755.504753:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760755.504761:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24682496, 24686592) append 0 append lockpos 0 00000080:00200000:1.0:1789760755.504772:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24682496=0x178a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760755.504969:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760755.504972:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24686592 00000020:00010000:1.0:1789760755.504981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760755.504984:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.504986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760755.504989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760755.504991:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760755.504998:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760755.505003:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760755.505017:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760755.505249:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24682496<>24686592 00000100:00100000:1.0:1789760755.505283:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760755.505297:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852800:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760755.560219:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973343 gen 2 00000008:00010000:0.0:1789760755.560259:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760755.560266:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760755.560275:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760755.560298:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760755.560323:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495852800:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760755.560389:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6026, 6026] 00000080:00200000:2.0:1789760755.560415:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.560421:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.560438:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24686592 i_blocks 49159 00000020:00200000:2.0:1789760755.560443:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.560458:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.560497:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24686592, count: 4096 00000080:00200000:2.0:1789760755.560516:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24686592, bytes: 4096 00000080:00200000:2.0:1789760755.560529:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.560538:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.560541:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24686592, 24690688) 0 0 00000080:00200000:2.0:1789760755.560546:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.560563:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6027, 6027] 00020000:00200000:2.0:1789760755.560571:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24686592 chunk: [24686592, 24690688) 24690688, 4096 00020000:00200000:2.0:1789760755.560575:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.560596:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6027, 6027] 00000008:00010000:2.0:1789760755.560615:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.560622:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760755.560634:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24686592, 24690688) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.560649:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24686592=0x178b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.560712:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.560715:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24690688 00000020:00010000:2.0:1789760755.560724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.560727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.560730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.560734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.560737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.560744:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.560751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.560771:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760755.560904:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 24686592<>24690688 00000100:00100000:3.0:1789760755.561016:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495852928:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760755.611402:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973344 gen 2 00000008:00010000:3.0:1789760755.611438:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760755.611446:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.611455:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.611480:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760755.611520:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495852928:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760755.611664:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6027, 6027] 00000080:00200000:2.0:1789760755.611676:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.611681:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.611694:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24690688 i_blocks 49159 00000020:00200000:2.0:1789760755.611697:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.611705:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.611731:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24690688, count: 4096 00000080:00200000:2.0:1789760755.611738:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24690688, bytes: 4096 00000080:00200000:2.0:1789760755.611745:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.611752:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.611755:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24690688, 24694784) 0 0 00000080:00200000:2.0:1789760755.611758:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.611771:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6028, 6028] 00020000:00200000:2.0:1789760755.611778:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24690688 chunk: [24690688, 24694784) 24694784, 4096 00020000:00200000:2.0:1789760755.611781:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.611795:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6028, 6028] 00000008:00010000:2.0:1789760755.611810:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.611815:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760755.611835:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24690688, 24694784) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.611845:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24690688=0x178c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.611899:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.611901:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24694784 00000020:00010000:2.0:1789760755.611909:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.611912:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.611915:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.611918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.611920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.611927:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.611932:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.611945:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760755.612039:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 24690688<>24694784 00000100:00100000:3.0:1789760755.612273:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760755.612288:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853056:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760755.646819:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973345 gen 2 00000008:00010000:3.0:1789760755.646874:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760755.646881:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.646889:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.646909:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760755.646932:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853056:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760755.647121:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6028, 6028] 00000080:00200000:2.0:1789760755.647135:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.647140:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.647153:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24694784 i_blocks 49159 00000020:00200000:2.0:1789760755.647157:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.647164:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.647214:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24694784, count: 4096 00000080:00200000:2.0:1789760755.647220:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24694784, bytes: 4096 00000080:00200000:2.0:1789760755.647227:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.647234:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.647237:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24694784, 24698880) 0 0 00000080:00200000:2.0:1789760755.647240:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.647253:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6029, 6029] 00020000:00200000:2.0:1789760755.647260:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24694784 chunk: [24694784, 24698880) 24698880, 4096 00020000:00200000:2.0:1789760755.647264:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.647277:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6029, 6029] 00000008:00010000:2.0:1789760755.647289:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.647294:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760755.647302:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24694784, 24698880) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.647311:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24694784=0x178d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.647349:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.647352:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24698880 00000020:00010000:2.0:1789760755.647358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.647362:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.647364:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.647367:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.647370:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.647375:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.647380:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.647392:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760755.647474:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 24694784<>24698880 00000100:00100000:3.0:1789760755.647575:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853184:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760755.682074:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760755.682086:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.682095:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.682117:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760755.682142:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853184:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760755.682250:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6029, 6029] 00000080:00200000:2.0:1789760755.682263:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.682268:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.682280:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24698880 i_blocks 49159 00000020:00200000:2.0:1789760755.682284:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.682291:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.682317:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24698880, count: 4096 00000080:00200000:2.0:1789760755.682323:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24698880, bytes: 4096 00000080:00200000:2.0:1789760755.682330:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.682337:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.682340:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24698880, 24702976) 0 0 00000080:00200000:2.0:1789760755.682344:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.682356:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6030, 6030] 00020000:00200000:2.0:1789760755.682362:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24698880 chunk: [24698880, 24702976) 24702976, 4096 00020000:00200000:2.0:1789760755.682366:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.682381:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6030, 6030] 00000008:00010000:2.0:1789760755.682394:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.682399:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760755.682407:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24698880, 24702976) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.682418:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24698880=0x178e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.682460:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.682463:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24702976 00000020:00010000:2.0:1789760755.682470:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.682473:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.682476:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.682479:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.682482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.682488:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.682493:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.682518:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760755.682613:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 24698880<>24702976 00000100:00100000:3.0:1789760755.682699:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760755.682712:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853312:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760755.719795:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973346 gen 2 00000100:00100000:3.0:1789760755.719831:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266300 x1876699495853312/t4294973347(4294973347) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760771 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760755.719878:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760755.719884:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.719892:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.719914:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760755.719934:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853312:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760755.720074:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6030, 6030] 00000080:00200000:2.0:1789760755.720088:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.720093:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.720104:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24702976 i_blocks 49159 00000020:00200000:2.0:1789760755.720107:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.720115:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.720140:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24702976, count: 4096 00000080:00200000:2.0:1789760755.720146:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24702976, bytes: 4096 00000080:00200000:2.0:1789760755.720153:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.720160:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.720163:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24702976, 24707072) 0 0 00000080:00200000:2.0:1789760755.720193:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.720207:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6031, 6031] 00020000:00200000:2.0:1789760755.720214:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24702976 chunk: [24702976, 24707072) 24707072, 4096 00020000:00200000:2.0:1789760755.720218:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.720234:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6031, 6031] 00000008:00010000:2.0:1789760755.720247:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.720252:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760755.720261:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24702976, 24707072) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.720271:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24702976=0x178f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.720315:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.720318:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24707072 00000020:00010000:2.0:1789760755.720325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.720329:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.720332:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.720335:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.720338:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.720344:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.720349:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.720362:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760755.720457:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 24702976<>24707072 00000100:00100000:3.0:1789760755.720570:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853440:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760755.766449:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973348 gen 2 00000008:00010000:3.0:1789760755.766498:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760755.766515:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.766524:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.766596:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760755.766621:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853440:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760755.768510:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6031, 6031] 00000080:00200000:2.0:1789760755.768531:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.768537:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.768552:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24707072 i_blocks 49159 00000020:00200000:2.0:1789760755.768557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.768570:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.768749:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24707072, count: 4096 00000080:00200000:2.0:1789760755.768758:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24707072, bytes: 4096 00000080:00200000:2.0:1789760755.768769:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.768777:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.768779:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24707072, 24711168) 0 0 00000080:00200000:2.0:1789760755.768783:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.768796:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6032, 6032] 00020000:00200000:2.0:1789760755.768803:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24707072 chunk: [24707072, 24711168) 24711168, 4096 00020000:00200000:2.0:1789760755.768807:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.768824:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6032, 6032] 00000008:00010000:2.0:1789760755.768838:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.768843:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760755.768916:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24707072, 24711168) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.768929:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24707072=0x1790000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.768986:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.768989:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24711168 00000020:00010000:2.0:1789760755.768996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.768999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.769002:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.769005:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.769008:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.769014:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.770794:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.770822:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760755.771369:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 24707072<>24711168 00000100:00100000:3.0:1789760755.771765:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495853568:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760755.907535:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760755.907546:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.907557:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.907582:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760755.907607:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495853568:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760755.907698:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6032, 6032] 00000080:00200000:2.0:1789760755.907711:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.907716:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.907728:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24711168 i_blocks 49159 00000020:00200000:2.0:1789760755.907731:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.907740:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.907765:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24711168, count: 4096 00000080:00200000:2.0:1789760755.907771:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24711168, bytes: 4096 00000080:00200000:2.0:1789760755.907778:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.907785:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.907788:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24711168, 24715264) 0 0 00000080:00200000:2.0:1789760755.907791:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.907804:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6033, 6033] 00020000:00200000:2.0:1789760755.907811:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24711168 chunk: [24711168, 24715264) 24715264, 4096 00020000:00200000:2.0:1789760755.907814:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.907839:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6033, 6033] 00000008:00010000:2.0:1789760755.907853:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.907858:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760755.907867:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24711168, 24715264) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.907878:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24711168=0x1791000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.907923:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.907926:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24715264 00000020:00010000:2.0:1789760755.907932:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.907935:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.907937:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.907940:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.907943:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.907949:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.907954:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.907968:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760755.908593:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 24711168<>24715264 00000100:00100000:3.0:1789760755.908658:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853696:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760755.960848:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973350 gen 2 00000008:00010000:3.0:1789760755.960899:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760755.960905:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.960914:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.960939:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760755.960967:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853696:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760755.961714:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6033, 6033] 00000080:00200000:0.0:1789760755.961729:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.961735:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.961748:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24715264 i_blocks 49159 00000020:00200000:0.0:1789760755.961752:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.961761:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.961787:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24715264, count: 4096 00000080:00200000:0.0:1789760755.961792:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24715264, bytes: 4096 00000080:00200000:0.0:1789760755.961802:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.961810:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.961812:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24715264, 24719360) 0 0 00000080:00200000:0.0:1789760755.961816:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.961828:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6034, 6034] 00020000:00200000:0.0:1789760755.961836:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24715264 chunk: [24715264, 24719360) 24719360, 4096 00020000:00200000:0.0:1789760755.961839:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.961855:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6034, 6034] 00000008:00010000:0.0:1789760755.961869:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.961875:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760755.961885:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24715264, 24719360) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.961896:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24715264=0x1792000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.961945:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.961947:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24719360 00000020:00010000:0.0:1789760755.961954:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.961957:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.961960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.961964:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.961967:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.961974:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.961980:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.961996:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760755.962098:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 24715264<>24719360 00000100:00100000:1.0:1789760755.962203:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495853824:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.048990:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973351 gen 2 00000008:00010000:0.0:1789760756.049030:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760756.049036:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.049046:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.049148:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760756.049205:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495853824:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760756.059610:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6034, 6034] 00000080:00200000:2.0:1789760756.059630:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.059635:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.059679:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24719360 i_blocks 49159 00000020:00200000:2.0:1789760756.059684:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.059697:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.059721:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24719360, count: 4096 00000080:00200000:2.0:1789760756.059728:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24719360, bytes: 4096 00000080:00200000:2.0:1789760756.059740:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.059748:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.059751:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24719360, 24723456) 0 0 00000080:00200000:2.0:1789760756.059755:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.059768:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6035, 6035] 00020000:00200000:2.0:1789760756.059775:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24719360 chunk: [24719360, 24723456) 24723456, 4096 00020000:00200000:2.0:1789760756.059780:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.059797:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6035, 6035] 00000008:00010000:2.0:1789760756.059811:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.059816:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760756.060378:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24719360, 24723456) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.060391:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24719360=0x1793000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.060520:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.060523:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24723456 00000020:00010000:2.0:1789760756.060531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.060533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.060536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.060539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.060542:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.060550:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.060556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.060573:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760756.060669:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 24719360<>24723456 00000100:00100000:2.0:1789760756.060719:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760756.060739:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853952:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760756.131228:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760756.131239:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760756.131249:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760756.131267:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760756.131290:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495853952:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760756.131440:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6035, 6035] 00000080:00200000:3.0:1789760756.131456:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760756.131461:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760756.131475:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24723456 i_blocks 49159 00000020:00200000:3.0:1789760756.131478:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760756.131486:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760756.131525:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24723456, count: 4096 00000080:00200000:3.0:1789760756.131531:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24723456, bytes: 4096 00000080:00200000:3.0:1789760756.131540:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760756.131547:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760756.131549:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24723456, 24727552) 0 0 00000080:00200000:3.0:1789760756.131552:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760756.131564:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6036, 6036] 00020000:00200000:3.0:1789760756.131571:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24723456 chunk: [24723456, 24727552) 24727552, 4096 00020000:00200000:3.0:1789760756.131575:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760756.131591:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6036, 6036] 00000008:00010000:3.0:1789760756.131606:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760756.131611:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760756.131620:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24723456, 24727552) append 0 append lockpos 0 00000080:00200000:3.0:1789760756.131631:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24723456=0x1794000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760756.131680:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760756.131682:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24727552 00000020:00010000:3.0:1789760756.131689:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760756.131692:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760756.131694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760756.131698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760756.131701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760756.131707:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760756.131711:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760756.131724:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760756.131815:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 24723456<>24727552 00000100:00100000:3.0:1789760756.131854:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760756.131866:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495854080:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760756.173622:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760756.221110:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973353 gen 2 00000008:00010000:3.0:1789760756.221200:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:3.0:1789760756.221207:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.221217:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.221242:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:3.0:1789760756.221266:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495854080:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760756.221414:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6036, 6036] 00000080:00200000:0.0:1789760756.221435:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.221440:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.221452:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24727552 i_blocks 49159 00000020:00200000:0.0:1789760756.221455:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.221464:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.221489:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24727552, count: 4096 00000080:00200000:0.0:1789760756.221495:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24727552, bytes: 4096 00000080:00200000:0.0:1789760756.221513:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.221522:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.221524:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24727552, 24731648) 0 0 00000080:00200000:0.0:1789760756.221528:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.221540:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6037, 6037] 00020000:00200000:0.0:1789760756.221548:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24727552 chunk: [24727552, 24731648) 24731648, 4096 00020000:00200000:0.0:1789760756.221552:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.221568:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6037, 6037] 00000008:00010000:0.0:1789760756.221584:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.221590:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760756.221599:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24727552, 24731648) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.221610:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24727552=0x1795000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.221657:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.221660:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24731648 00000020:00010000:0.0:1789760756.221667:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.221670:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.221673:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.221676:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.221679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.221685:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.221690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.221705:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760756.221800:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 24727552<>24731648 00000100:00100000:0.0:1789760756.222034:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760756.222049:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495854208:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760756.314097:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760756.314108:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.314118:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.314137:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760756.314164:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495854208:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.314293:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6037, 6037] 00000080:00200000:1.0:1789760756.314306:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.314312:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.314330:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24731648 i_blocks 49159 00000020:00200000:1.0:1789760756.314334:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.314342:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.314367:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24731648, count: 4096 00000080:00200000:1.0:1789760756.314373:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24731648, bytes: 4096 00000080:00200000:1.0:1789760756.314381:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.314397:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.314398:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24731648, 24735744) 0 0 00000080:00200000:1.0:1789760756.314402:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.314416:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6038, 6038] 00020000:00200000:1.0:1789760756.314423:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24731648 chunk: [24731648, 24735744) 24735744, 4096 00020000:00200000:1.0:1789760756.314427:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.314442:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6038, 6038] 00000008:00010000:1.0:1789760756.314456:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.314461:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.314470:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24731648, 24735744) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.314480:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24731648=0x1796000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.314535:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.314538:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24735744 00000020:00010000:1.0:1789760756.314556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.314560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.314563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.314566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.314569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.314575:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.314581:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.314597:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760756.314700:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 24731648<>24735744 00000100:00100000:1.0:1789760756.314742:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760756.314756:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495854336:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.377091:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973355 gen 2 00000008:00010000:0.0:1789760756.377139:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760756.377146:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.377154:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.377203:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760756.377227:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495854336:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.377317:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6038, 6038] 00000080:00200000:1.0:1789760756.377330:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.377335:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.377348:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24735744 i_blocks 49159 00000020:00200000:1.0:1789760756.377351:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.377359:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.377555:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24735744, count: 4096 00000080:00200000:1.0:1789760756.377562:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24735744, bytes: 4096 00000080:00200000:1.0:1789760756.377571:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.377578:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.377580:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24735744, 24739840) 0 0 00000080:00200000:1.0:1789760756.377584:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.377596:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6039, 6039] 00020000:00200000:1.0:1789760756.377603:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24735744 chunk: [24735744, 24739840) 24739840, 4096 00020000:00200000:1.0:1789760756.377607:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.377623:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6039, 6039] 00000008:00010000:1.0:1789760756.377637:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.377642:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.377652:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24735744, 24739840) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.377662:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24735744=0x1797000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.377711:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.377713:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24739840 00000020:00010000:1.0:1789760756.377719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.377722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.377724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.377727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.377730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.377735:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.377741:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.377754:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760756.377987:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 24735744<>24739840 00000100:00100000:0.0:1789760756.378057:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495854464:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.416222:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973356 gen 2 00000008:00010000:0.0:1789760756.416262:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760756.416268:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.416277:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.416298:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760756.416322:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495854464:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.416414:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6039, 6039] 00000080:00200000:1.0:1789760756.416428:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.416434:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.416446:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24739840 i_blocks 49159 00000020:00200000:1.0:1789760756.416450:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.416458:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.416503:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24739840, count: 4096 00000080:00200000:1.0:1789760756.416510:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24739840, bytes: 4096 00000080:00200000:1.0:1789760756.416519:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.416526:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.416528:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24739840, 24743936) 0 0 00000080:00200000:1.0:1789760756.416532:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.416545:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6040, 6040] 00020000:00200000:1.0:1789760756.416552:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24739840 chunk: [24739840, 24743936) 24743936, 4096 00020000:00200000:1.0:1789760756.416556:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.416572:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6040, 6040] 00000008:00010000:1.0:1789760756.416587:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.416592:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.416601:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24739840, 24743936) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.416612:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24739840=0x1798000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.416658:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.416661:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24743936 00000020:00010000:1.0:1789760756.416669:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.416672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.416675:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.416678:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.416681:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.416687:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.416693:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.416708:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760756.416807:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 24739840<>24743936 00000100:00100000:0.0:1789760756.417139:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495854592:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.456381:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973357 gen 2 00000008:00010000:0.0:1789760756.456427:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760756.456435:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.456445:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.456465:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760756.456564:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495854592:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.456764:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6040, 6040] 00000080:00200000:1.0:1789760756.456777:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.456782:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.456794:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24743936 i_blocks 49159 00000020:00200000:1.0:1789760756.456797:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.456805:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.456840:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24743936, count: 4096 00000080:00200000:1.0:1789760756.456847:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24743936, bytes: 4096 00000080:00200000:1.0:1789760756.456865:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.456873:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.456875:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24743936, 24748032) 0 0 00000080:00200000:1.0:1789760756.456879:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.456892:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6041, 6041] 00020000:00200000:1.0:1789760756.456899:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24743936 chunk: [24743936, 24748032) 24748032, 4096 00020000:00200000:1.0:1789760756.456904:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.456919:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6041, 6041] 00000008:00010000:1.0:1789760756.456932:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.456938:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.456947:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24743936, 24748032) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.456958:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24743936=0x1799000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.457005:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.457007:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24748032 00000020:00010000:1.0:1789760756.457015:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.457019:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.457022:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.457025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.457029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.457035:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.457041:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.457057:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760756.457353:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 24743936<>24748032 00000100:00100000:0.0:1789760756.457445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495854720:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.494690:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973358 gen 2 00000008:00010000:0.0:1789760756.494728:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760756.494735:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.494743:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.494762:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760756.494786:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495854720:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.495112:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6041, 6041] 00000080:00200000:1.0:1789760756.495126:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.495132:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.495144:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24748032 i_blocks 49159 00000020:00200000:1.0:1789760756.495148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.495156:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.495207:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24748032, count: 4096 00000080:00200000:1.0:1789760756.495213:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24748032, bytes: 4096 00000080:00200000:1.0:1789760756.495221:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.495228:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.495231:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24748032, 24752128) 0 0 00000080:00200000:1.0:1789760756.495234:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.495247:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6042, 6042] 00020000:00200000:1.0:1789760756.495255:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24748032 chunk: [24748032, 24752128) 24752128, 4096 00020000:00200000:1.0:1789760756.495259:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.495275:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6042, 6042] 00000008:00010000:1.0:1789760756.495291:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.495296:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.495305:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24748032, 24752128) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.495317:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24748032=0x179a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.495365:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.495368:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24752128 00000020:00010000:1.0:1789760756.495374:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.495378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.495381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.495384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.495386:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.495392:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.495398:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.495413:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760756.495585:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 24748032<>24752128 00000100:00100000:0.0:1789760756.495650:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495854848:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.555049:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973359 gen 2 00000008:00010000:0.0:1789760756.555448:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760756.555457:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.555469:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.555722:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760756.555754:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495854848:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.555845:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6042, 6042] 00000080:00200000:1.0:1789760756.555865:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.555871:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.555889:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24752128 i_blocks 49159 00000020:00200000:1.0:1789760756.555893:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.555903:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.555938:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24752128, count: 4096 00000080:00200000:1.0:1789760756.555947:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24752128, bytes: 4096 00000080:00200000:1.0:1789760756.555960:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.555970:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.555973:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24752128, 24756224) 0 0 00000080:00200000:1.0:1789760756.555978:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.555996:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6043, 6043] 00020000:00200000:1.0:1789760756.556005:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24752128 chunk: [24752128, 24756224) 24756224, 4096 00020000:00200000:1.0:1789760756.556011:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.556033:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6043, 6043] 00000008:00010000:1.0:1789760756.556053:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.556445:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.556461:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24752128, 24756224) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.556475:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24752128=0x179b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.556759:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.556763:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24756224 00000020:00010000:1.0:1789760756.556772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.556776:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.556779:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.556783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.556786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.556796:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.556803:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.556828:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760756.556962:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 24752128<>24756224 00000100:00100000:0.0:1789760756.557056:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495854976:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.607049:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973360 gen 2 00000008:00010000:0.0:1789760756.607088:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760756.607094:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.607103:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.607215:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760756.607241:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495854976:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.607384:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6043, 6043] 00000080:00200000:1.0:1789760756.607397:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.607403:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.607414:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24756224 i_blocks 49159 00000020:00200000:1.0:1789760756.607417:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.607425:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.607449:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24756224, count: 4096 00000080:00200000:1.0:1789760756.607455:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24756224, bytes: 4096 00000080:00200000:1.0:1789760756.607463:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.607470:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.607602:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24756224, 24760320) 0 0 00000080:00200000:1.0:1789760756.607607:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.607620:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6044, 6044] 00020000:00200000:1.0:1789760756.607628:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24756224 chunk: [24756224, 24760320) 24760320, 4096 00020000:00200000:1.0:1789760756.607632:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.607648:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6044, 6044] 00000008:00010000:1.0:1789760756.607662:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.607666:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.607675:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24756224, 24760320) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.607687:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24756224=0x179c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.607733:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.607736:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24760320 00000020:00010000:1.0:1789760756.607743:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.607747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.607750:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.607753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.607755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.607761:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.607767:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.607783:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760756.608061:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 24756224<>24760320 00000100:00100000:0.0:1789760756.608126:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495855104:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.663918:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973361 gen 2 00000008:00010000:0.0:1789760756.663968:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760756.663975:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.663984:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.664003:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760756.664028:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495855104:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.664257:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6044, 6044] 00000080:00200000:1.0:1789760756.664270:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.664275:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.664287:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24760320 i_blocks 49159 00000020:00200000:1.0:1789760756.664291:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.664299:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.664325:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24760320, count: 4096 00000080:00200000:1.0:1789760756.664331:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24760320, bytes: 4096 00000080:00200000:1.0:1789760756.664338:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.664345:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.664348:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24760320, 24764416) 0 0 00000080:00200000:1.0:1789760756.664351:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.664365:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6045, 6045] 00020000:00200000:1.0:1789760756.664372:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24760320 chunk: [24760320, 24764416) 24764416, 4096 00020000:00200000:1.0:1789760756.664376:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.664393:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6045, 6045] 00000008:00010000:1.0:1789760756.664407:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.664412:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.664422:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24760320, 24764416) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.664433:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24760320=0x179d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.664473:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.664475:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24764416 00000020:00010000:1.0:1789760756.664482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.664484:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.664487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.664490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.664504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.664511:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.664516:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.664530:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760756.664622:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 24760320<>24764416 00000100:00100000:0.0:1789760756.664883:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495855232:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.698473:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973362 gen 2 00000008:00010000:0.0:1789760756.698518:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760756.698525:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.698533:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.698552:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760756.698581:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495855232:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.698683:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6045, 6045] 00000080:00200000:1.0:1789760756.698697:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.698703:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.698715:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24764416 i_blocks 49159 00000020:00200000:1.0:1789760756.698719:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.698727:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.698751:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24764416, count: 4096 00000080:00200000:1.0:1789760756.698758:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24764416, bytes: 4096 00000080:00200000:1.0:1789760756.698765:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.698772:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.698774:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24764416, 24768512) 0 0 00000080:00200000:1.0:1789760756.698777:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.698790:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6046, 6046] 00020000:00200000:1.0:1789760756.698796:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24764416 chunk: [24764416, 24768512) 24768512, 4096 00020000:00200000:1.0:1789760756.698801:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.698817:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6046, 6046] 00000008:00010000:1.0:1789760756.698831:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.698836:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.698845:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24764416, 24768512) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.698855:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24764416=0x179e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.698907:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.698910:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24768512 00000020:00010000:1.0:1789760756.698918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.698921:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.698924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.698927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.698930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.698936:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.698941:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.698957:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760756.699059:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 24764416<>24768512 00000100:00100000:0.0:1789760756.699438:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760756.699454:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495855360:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.749278:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973363 gen 2 00000008:00010000:0.0:1789760756.749312:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760756.749318:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.749327:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.749394:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760756.749418:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495855360:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.761751:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6046, 6046] 00000080:00200000:1.0:1789760756.761774:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.761780:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.761794:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24768512 i_blocks 49159 00000020:00200000:1.0:1789760756.761799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.761811:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.761839:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24768512, count: 4096 00000080:00200000:1.0:1789760756.761846:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24768512, bytes: 4096 00000080:00200000:1.0:1789760756.761856:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.761864:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.761866:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24768512, 24772608) 0 0 00000080:00200000:1.0:1789760756.761870:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.761884:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6047, 6047] 00020000:00200000:1.0:1789760756.761892:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24768512 chunk: [24768512, 24772608) 24772608, 4096 00020000:00200000:1.0:1789760756.761897:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.761914:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6047, 6047] 00000008:00010000:1.0:1789760756.761929:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.761934:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.761943:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24768512, 24772608) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.761954:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24768512=0x179f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.762010:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.762013:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24772608 00000020:00010000:1.0:1789760756.762019:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.762023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.762026:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.762029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.762032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.762040:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.762046:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.762071:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760756.762203:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 24768512<>24772608 00000100:00100000:0.0:1789760756.762397:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495855488:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760756.827022:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760756.827032:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.827040:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.827060:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760756.827086:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495855488:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.827475:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6047, 6047] 00000080:00200000:1.0:1789760756.827489:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.827505:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.827516:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24772608 i_blocks 49159 00000020:00200000:1.0:1789760756.827520:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.827528:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.827553:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24772608, count: 4096 00000080:00200000:1.0:1789760756.827560:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24772608, bytes: 4096 00000080:00200000:1.0:1789760756.827568:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.827691:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.827694:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24772608, 24776704) 0 0 00000080:00200000:1.0:1789760756.827699:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.827712:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6048, 6048] 00020000:00200000:1.0:1789760756.827721:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24772608 chunk: [24772608, 24776704) 24776704, 4096 00020000:00200000:1.0:1789760756.827725:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.827741:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6048, 6048] 00000008:00010000:1.0:1789760756.827756:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.827763:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.827771:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24772608, 24776704) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.827782:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24772608=0x17a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.827828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.827831:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24776704 00000020:00010000:1.0:1789760756.827837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.827851:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.827855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.827858:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.827860:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.827867:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.827873:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.827888:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760756.827978:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 24772608<>24776704 00000100:00100000:0.0:1789760756.828318:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495855616:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760756.911301:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973364 gen 2 00000100:00100000:0.0:1789760756.911331:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504ea00 x1876699495855616/t4294973365(4294973365) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760772 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760756.911368:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760756.911374:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.911382:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.911400:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760756.911420:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495855616:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760756.911519:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6048, 6048] 00000080:00200000:1.0:1789760756.911531:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.911536:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.911548:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24776704 i_blocks 49159 00000020:00200000:1.0:1789760756.911551:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.911558:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.911583:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24776704, count: 4096 00000080:00200000:1.0:1789760756.911589:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24776704, bytes: 4096 00000080:00200000:1.0:1789760756.911597:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.911604:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.911606:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24776704, 24780800) 0 0 00000080:00200000:1.0:1789760756.911610:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.911622:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6049, 6049] 00020000:00200000:1.0:1789760756.911629:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24776704 chunk: [24776704, 24780800) 24780800, 4096 00020000:00200000:1.0:1789760756.911632:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.911647:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6049, 6049] 00000008:00010000:1.0:1789760756.911660:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.911664:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760756.911674:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24776704, 24780800) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.911684:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24776704=0x17a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.911740:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.911743:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24780800 00000020:00010000:1.0:1789760756.911749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.911752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.911755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.911758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.911761:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.911767:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.911771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.911784:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760756.911995:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 24776704<>24780800 00000100:00100000:0.0:1789760756.912145:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495855744:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.006720:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973366 gen 2 00000008:00010000:0.0:1789760757.006768:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760757.006774:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.006793:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.006809:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760757.006833:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495855744:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.006967:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6049, 6049] 00000080:00200000:1.0:1789760757.006979:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.006984:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.006995:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24780800 i_blocks 49159 00000020:00200000:1.0:1789760757.006998:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.007006:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.007029:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24780800, count: 4096 00000080:00200000:1.0:1789760757.007035:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24780800, bytes: 4096 00000080:00200000:1.0:1789760757.007041:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.007048:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.007050:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24780800, 24784896) 0 0 00000080:00200000:1.0:1789760757.007054:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.007077:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6050, 6050] 00020000:00200000:1.0:1789760757.007084:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24780800 chunk: [24780800, 24784896) 24784896, 4096 00020000:00200000:1.0:1789760757.007088:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.007103:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6050, 6050] 00000008:00010000:1.0:1789760757.007117:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.007122:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.007130:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24780800, 24784896) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.007235:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24780800=0x17a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.007280:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.007282:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24784896 00000020:00010000:1.0:1789760757.007289:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.007292:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.007294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.007298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.007300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.007306:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.007311:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.007325:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760757.007412:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 24780800<>24784896 00000100:00100000:0.0:1789760757.007630:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495855872:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.072776:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973367 gen 2 00000008:00010000:0.0:1789760757.072814:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760757.072821:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.072829:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.072847:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760757.072871:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495855872:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.072959:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6050, 6050] 00000080:00200000:1.0:1789760757.072972:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.072977:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.072989:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24784896 i_blocks 49159 00000020:00200000:1.0:1789760757.072992:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.073000:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.073022:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24784896, count: 4096 00000080:00200000:1.0:1789760757.073027:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24784896, bytes: 4096 00000080:00200000:1.0:1789760757.073034:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.073040:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.073042:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24784896, 24788992) 0 0 00000080:00200000:1.0:1789760757.073046:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.073058:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6051, 6051] 00020000:00200000:1.0:1789760757.074186:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24784896 chunk: [24784896, 24788992) 24788992, 4096 00020000:00200000:1.0:1789760757.074194:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.074214:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6051, 6051] 00000008:00010000:1.0:1789760757.074229:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.074235:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.074245:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24784896, 24788992) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.074257:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24784896=0x17a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.074306:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.074309:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24788992 00000020:00010000:1.0:1789760757.074317:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.074320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.074323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.074326:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.074329:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.074336:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.074342:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.074362:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760757.074460:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 24784896<>24788992 00000100:00100000:0.0:1789760757.075257:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495856000:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.129242:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973368 gen 2 00000008:00010000:0.0:1789760757.129281:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760757.129288:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.129297:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.129316:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760757.129339:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495856000:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.129427:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6051, 6051] 00000080:00200000:1.0:1789760757.129440:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.129445:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.129456:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24788992 i_blocks 49159 00000020:00200000:1.0:1789760757.129460:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.129467:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.129571:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24788992, count: 4096 00000080:00200000:1.0:1789760757.129579:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24788992, bytes: 4096 00000080:00200000:1.0:1789760757.129586:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.129593:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.129595:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24788992, 24793088) 0 0 00000080:00200000:1.0:1789760757.129599:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.129610:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6052, 6052] 00020000:00200000:1.0:1789760757.129618:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24788992 chunk: [24788992, 24793088) 24793088, 4096 00020000:00200000:1.0:1789760757.129622:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.129636:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6052, 6052] 00000008:00010000:1.0:1789760757.129650:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.129655:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.129665:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24788992, 24793088) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.129675:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24788992=0x17a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.129720:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.129723:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24793088 00000020:00010000:1.0:1789760757.129731:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.129734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.129737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.129741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.129744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.129750:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.129756:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.129770:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760757.129998:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 24788992<>24793088 00000100:00100000:0.0:1789760757.130221:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856128:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760757.197514:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:0.0:1789760757.201727:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760757.201743:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.201752:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.201777:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760757.201801:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856128:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.202198:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6052, 6052] 00000080:00200000:1.0:1789760757.202210:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.202216:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.202227:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24793088 i_blocks 49159 00000020:00200000:1.0:1789760757.202230:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.202238:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.202263:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24793088, count: 4096 00000080:00200000:1.0:1789760757.202268:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24793088, bytes: 4096 00000080:00200000:1.0:1789760757.202275:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.202289:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.202291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24793088, 24797184) 0 0 00000080:00200000:1.0:1789760757.202294:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.202306:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6053, 6053] 00020000:00200000:1.0:1789760757.202312:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24793088 chunk: [24793088, 24797184) 24797184, 4096 00020000:00200000:1.0:1789760757.202317:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.202447:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6053, 6053] 00000008:00010000:1.0:1789760757.202464:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.202469:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.202478:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24793088, 24797184) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.202498:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24793088=0x17a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.202605:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.202607:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24797184 00000020:00010000:1.0:1789760757.202614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.202617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.202621:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.202624:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.202626:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.202632:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.202638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.202653:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760757.203313:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 24793088<>24797184 00000100:00100000:0.0:1789760757.203418:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495856256:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.240390:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973369 gen 2 00000100:00100000:0.0:1789760757.240420:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504ca80 x1876699495856256/t4294973370(4294973370) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760773 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760757.240455:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760757.240461:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.240469:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.240541:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760757.240564:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495856256:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.245711:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6053, 6053] 00000080:00200000:1.0:1789760757.245732:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.245738:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.245752:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24797184 i_blocks 49159 00000020:00200000:1.0:1789760757.245756:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.245770:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.245794:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24797184, count: 4096 00000080:00200000:1.0:1789760757.245811:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24797184, bytes: 4096 00000080:00200000:1.0:1789760757.245821:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.245829:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.245831:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24797184, 24801280) 0 0 00000080:00200000:1.0:1789760757.245835:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.245848:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6054, 6054] 00020000:00200000:1.0:1789760757.245855:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24797184 chunk: [24797184, 24801280) 24801280, 4096 00020000:00200000:1.0:1789760757.245859:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.245877:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6054, 6054] 00000008:00010000:1.0:1789760757.245900:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.245905:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.245914:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24797184, 24801280) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.245926:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24797184=0x17a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.245991:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.245994:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24801280 00000020:00010000:1.0:1789760757.246000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.246004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.246007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.246010:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.246012:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.246019:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.246025:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.246047:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760757.246162:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 24797184<>24801280 00000100:00100000:0.0:1789760757.246467:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856384:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.277545:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973371 gen 2 00000008:00010000:0.0:1789760757.277609:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760757.277618:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.277628:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.277648:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760757.277676:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856384:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.277842:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6054, 6054] 00000080:00200000:1.0:1789760757.277858:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.277864:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.277880:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24801280 i_blocks 49159 00000020:00200000:1.0:1789760757.277884:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.277894:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.277990:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24801280, count: 4096 00000080:00200000:1.0:1789760757.277999:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24801280, bytes: 4096 00000080:00200000:1.0:1789760757.278009:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.278019:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.278021:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24801280, 24805376) 0 0 00000080:00200000:1.0:1789760757.278026:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.278044:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6055, 6055] 00020000:00200000:1.0:1789760757.278053:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24801280 chunk: [24801280, 24805376) 24805376, 4096 00020000:00200000:1.0:1789760757.278058:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.278080:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6055, 6055] 00000008:00010000:1.0:1789760757.278356:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.278362:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.278374:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24801280, 24805376) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.278388:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24801280=0x17a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.278445:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.278448:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24805376 00000020:00010000:1.0:1789760757.279767:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.279772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.279777:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.279782:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.279785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.279797:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.279805:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.279833:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760757.280141:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 24801280<>24805376 00000100:00100000:0.0:1789760757.280439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495856512:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.318690:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973372 gen 2 00000008:00010000:0.0:1789760757.318744:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760757.318753:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.318765:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.319436:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760757.319471:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495856512:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.320337:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6055, 6055] 00000080:00200000:1.0:1789760757.320358:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.320364:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.320382:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24805376 i_blocks 49159 00000020:00200000:1.0:1789760757.320387:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.320399:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.320435:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24805376, count: 4096 00000080:00200000:1.0:1789760757.320451:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24805376, bytes: 4096 00000080:00200000:1.0:1789760757.320466:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.320584:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.320866:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24805376, 24809472) 0 0 00000080:00200000:1.0:1789760757.320872:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.320893:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6056, 6056] 00020000:00200000:1.0:1789760757.320901:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24805376 chunk: [24805376, 24809472) 24809472, 4096 00020000:00200000:1.0:1789760757.320905:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.320927:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6056, 6056] 00000008:00010000:1.0:1789760757.320945:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.320951:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.320964:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24805376, 24809472) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.320978:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24805376=0x17a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.321037:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.321039:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24809472 00000020:00010000:1.0:1789760757.321047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.321051:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.321054:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.321057:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.321060:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.321068:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.321074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.321097:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760757.321239:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 24805376<>24809472 00000100:00100000:0.0:1789760757.321769:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856640:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.362526:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973373 gen 2 00000008:00010000:0.0:1789760757.362577:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760757.362586:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.362598:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.362667:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760757.362697:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856640:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.366760:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6056, 6056] 00000080:00200000:1.0:1789760757.366787:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.366793:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.366812:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24809472 i_blocks 49159 00000020:00200000:1.0:1789760757.366818:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.366831:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.366863:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24809472, count: 4096 00000080:00200000:1.0:1789760757.366871:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24809472, bytes: 4096 00000080:00200000:1.0:1789760757.366883:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.366891:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.366893:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24809472, 24813568) 0 0 00000080:00200000:1.0:1789760757.366897:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.366914:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6057, 6057] 00020000:00200000:1.0:1789760757.366923:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24809472 chunk: [24809472, 24813568) 24813568, 4096 00020000:00200000:1.0:1789760757.366927:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.366950:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6057, 6057] 00000008:00010000:1.0:1789760757.366968:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.366974:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.366985:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24809472, 24813568) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.367152:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24809472=0x17a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.367241:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.367245:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24813568 00000020:00010000:1.0:1789760757.367258:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.367262:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.367266:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.367270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.367273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.367280:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.367286:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.367316:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760757.367668:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 24809472<>24813568 00000100:00100000:0.0:1789760757.367891:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760757.367912:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856768:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.426351:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973374 gen 2 00000008:00010000:0.0:1789760757.426412:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760757.426420:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.426431:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.426450:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000080:00200000:1.0:1789760757.426602:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6057, 6057] 00000080:00200000:1.0:1789760757.426622:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.426629:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.426646:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24813568 i_blocks 49159 00000020:00200000:1.0:1789760757.426650:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.426659:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.426692:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24813568, count: 4096 00000080:00200000:1.0:1789760757.426700:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24813568, bytes: 4096 00000080:00200000:1.0:1789760757.426710:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.426719:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.426722:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24813568, 24817664) 0 0 00000080:00200000:1.0:1789760757.426726:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.426741:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6058, 6058] 00020000:00200000:1.0:1789760757.426749:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24813568 chunk: [24813568, 24817664) 24817664, 4096 00020000:00200000:1.0:1789760757.426754:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.426772:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6058, 6058] 00000008:00010000:1.0:1789760757.426789:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.426794:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.426805:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24813568, 24817664) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.426829:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24813568=0x17aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.426889:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.426892:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24817664 00000020:00010000:1.0:1789760757.426900:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.426903:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.426906:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.426910:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.426912:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.426920:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.426927:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1789760757.427067:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856768:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760757.427109:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:0.0:1789760757.427283:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 24813568<>24817664 00000100:00100000:0.0:1789760757.427337:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856896:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.464572:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973375 gen 2 00000008:00010000:0.0:1789760757.464621:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760757.464629:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.464639:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.464700:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760757.464728:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495856896:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.465028:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6058, 6058] 00000080:00200000:1.0:1789760757.465045:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.465052:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.465069:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24817664 i_blocks 49159 00000020:00200000:1.0:1789760757.465072:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.465081:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.465114:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24817664, count: 4096 00000080:00200000:1.0:1789760757.465121:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24817664, bytes: 4096 00000080:00200000:1.0:1789760757.465132:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.465140:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.465142:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24817664, 24821760) 0 0 00000080:00200000:1.0:1789760757.465147:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.465162:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6059, 6059] 00020000:00200000:1.0:1789760757.465203:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24817664 chunk: [24817664, 24821760) 24821760, 4096 00020000:00200000:1.0:1789760757.465271:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.465294:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6059, 6059] 00000008:00010000:1.0:1789760757.465312:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.465317:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.465329:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24817664, 24821760) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.465342:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24817664=0x17ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.465463:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.465527:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24821760 00000020:00010000:1.0:1789760757.465537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.465540:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.465543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.465547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.465549:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.465559:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.465565:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.465584:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760757.465712:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 24817664<>24821760 00000100:00100000:0.0:1789760757.465784:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495857024:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.502029:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973376 gen 2 00000008:00010000:0.0:1789760757.502080:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760757.502088:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.502099:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.502315:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760757.502341:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495857024:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.502544:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6059, 6059] 00000080:00200000:1.0:1789760757.502560:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.502567:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.502584:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24821760 i_blocks 49159 00000020:00200000:1.0:1789760757.502588:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.502597:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.502630:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24821760, count: 4096 00000080:00200000:1.0:1789760757.502636:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24821760, bytes: 4096 00000080:00200000:1.0:1789760757.502646:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.502654:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.502656:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24821760, 24825856) 0 0 00000080:00200000:1.0:1789760757.502660:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.502676:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6060, 6060] 00020000:00200000:1.0:1789760757.502684:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24821760 chunk: [24821760, 24825856) 24825856, 4096 00020000:00200000:1.0:1789760757.502688:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.502706:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6060, 6060] 00000008:00010000:1.0:1789760757.502722:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.502728:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.502740:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24821760, 24825856) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.502754:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24821760=0x17ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.502806:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.502915:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24825856 00000020:00010000:1.0:1789760757.502926:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.502929:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.502933:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.502936:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.502939:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.502946:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.502952:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.502969:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760757.503088:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 24821760<>24825856 00000100:00100000:0.0:1789760757.503571:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495857152:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760757.537635:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760757.537650:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.537662:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.537741:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760757.537773:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495857152:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.537972:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6060, 6060] 00000080:00200000:1.0:1789760757.537989:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.537996:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.538011:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24825856 i_blocks 49159 00000020:00200000:1.0:1789760757.538065:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.538076:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.538112:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24825856, count: 4096 00000080:00200000:1.0:1789760757.538119:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24825856, bytes: 4096 00000080:00200000:1.0:1789760757.538133:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.538226:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.538228:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24825856, 24829952) 0 0 00000080:00200000:1.0:1789760757.538234:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.538251:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6061, 6061] 00020000:00200000:1.0:1789760757.538258:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24825856 chunk: [24825856, 24829952) 24829952, 4096 00020000:00200000:1.0:1789760757.538263:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.538283:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6061, 6061] 00000008:00010000:1.0:1789760757.538301:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.538307:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.538319:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24825856, 24829952) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.538333:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24825856=0x17ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.538386:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.538389:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24829952 00000020:00010000:1.0:1789760757.538396:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.538400:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.538403:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.538406:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.538410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.538417:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.538423:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.538441:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760757.538705:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 24825856<>24829952 00000100:00100000:0.0:1789760757.538779:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495857280:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.580126:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973377 gen 2 00000100:00100000:0.0:1789760757.580224:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504ce00 x1876699495857280/t4294973378(4294973378) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760773 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760757.580276:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760757.580283:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.580293:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.580323:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760757.580346:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495857280:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.580509:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6061, 6061] 00000080:00200000:1.0:1789760757.580525:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.580532:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.580547:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24829952 i_blocks 49159 00000020:00200000:1.0:1789760757.580551:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.580561:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.580598:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24829952, count: 4096 00000080:00200000:1.0:1789760757.580605:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24829952, bytes: 4096 00000080:00200000:1.0:1789760757.580619:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.580627:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.580629:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24829952, 24834048) 0 0 00000080:00200000:1.0:1789760757.580633:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.580651:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6062, 6062] 00020000:00200000:1.0:1789760757.580670:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24829952 chunk: [24829952, 24834048) 24834048, 4096 00020000:00200000:1.0:1789760757.580676:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.580697:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6062, 6062] 00000008:00010000:1.0:1789760757.580715:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.580721:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.580733:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24829952, 24834048) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.580747:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24829952=0x17ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.580805:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.580808:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24834048 00000020:00010000:1.0:1789760757.580817:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.580820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.580824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.580828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.580831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.580838:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.580845:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.580863:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760757.581005:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 24829952<>24834048 00000100:00100000:0.0:1789760757.581771:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495857408:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.613401:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973379 gen 2 00000008:00010000:0.0:1789760757.613464:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760757.613472:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.613482:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.613513:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760757.613546:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495857408:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.613770:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6062, 6062] 00000080:00200000:1.0:1789760757.613786:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.613794:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.613811:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24834048 i_blocks 49159 00000020:00200000:1.0:1789760757.613815:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.613824:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.613860:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24834048, count: 4096 00000080:00200000:1.0:1789760757.613868:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24834048, bytes: 4096 00000080:00200000:1.0:1789760757.613880:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.613889:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.613892:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24834048, 24838144) 0 0 00000080:00200000:1.0:1789760757.613896:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.613914:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6063, 6063] 00020000:00200000:1.0:1789760757.613922:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24834048 chunk: [24834048, 24838144) 24838144, 4096 00020000:00200000:1.0:1789760757.613927:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.613958:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6063, 6063] 00000008:00010000:1.0:1789760757.613975:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.613981:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.613992:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24834048, 24838144) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.614006:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24834048=0x17af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.614071:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.614074:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24838144 00000020:00010000:1.0:1789760757.614083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.614087:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.614089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.614093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.614095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.614102:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.614108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.614125:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760757.614289:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 24834048<>24838144 00000100:00100000:0.0:1789760757.614609:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760757.614757:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495857536:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.661261:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973380 gen 2 00000008:00010000:0.0:1789760757.661304:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760757.661312:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.661322:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.661368:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760757.661395:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495857536:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.661535:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6063, 6063] 00000080:00200000:1.0:1789760757.661600:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.661608:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.661686:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24838144 i_blocks 49159 00000020:00200000:1.0:1789760757.661690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.661700:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.661736:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24838144, count: 4096 00000080:00200000:1.0:1789760757.661742:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24838144, bytes: 4096 00000080:00200000:1.0:1789760757.661752:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.661761:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.661763:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24838144, 24842240) 0 0 00000080:00200000:1.0:1789760757.661768:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.661784:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6064, 6064] 00020000:00200000:1.0:1789760757.661792:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24838144 chunk: [24838144, 24842240) 24842240, 4096 00020000:00200000:1.0:1789760757.661797:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.661816:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6064, 6064] 00000008:00010000:1.0:1789760757.661834:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.661840:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.661852:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24838144, 24842240) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.661866:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24838144=0x17b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.661921:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.661924:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24842240 00000020:00010000:1.0:1789760757.661933:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.661936:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.661939:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.661943:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.661946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.661954:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.661960:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.661976:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760757.662165:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 24838144<>24842240 00000100:00100000:0.0:1789760757.662395:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495857664:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.698035:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973381 gen 2 00000008:00010000:0.0:1789760757.698092:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760757.698101:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.698111:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.698217:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760757.698246:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495857664:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760757.698341:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6064, 6064] 00000080:00200000:3.0:1789760757.698359:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760757.698365:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760757.698381:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24842240 i_blocks 49159 00000020:00200000:3.0:1789760757.698385:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760757.698394:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760757.698428:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24842240, count: 4096 00000080:00200000:3.0:1789760757.698435:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24842240, bytes: 4096 00000080:00200000:3.0:1789760757.698445:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760757.698465:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760757.698468:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24842240, 24846336) 0 0 00000080:00200000:3.0:1789760757.698473:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760757.698665:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6065, 6065] 00020000:00200000:3.0:1789760757.698673:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24842240 chunk: [24842240, 24846336) 24846336, 4096 00020000:00200000:3.0:1789760757.698679:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760757.698699:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6065, 6065] 00000008:00010000:3.0:1789760757.698716:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760757.698721:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760757.698735:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24842240, 24846336) append 0 append lockpos 0 00000080:00200000:3.0:1789760757.698749:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24842240=0x17b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760757.698890:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760757.698893:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24846336 00000020:00010000:3.0:1789760757.698903:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760757.698906:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.698909:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.698913:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760757.698915:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760757.698924:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760757.698932:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760757.698951:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760757.699083:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 24842240<>24846336 00000100:00100000:2.0:1789760757.699220:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760757.699241:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495857792:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760757.739315:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973382 gen 2 00000008:00010000:2.0:1789760757.739362:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760757.739370:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760757.739381:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760757.739401:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760757.739428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495857792:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760757.739852:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6065, 6065] 00000080:00200000:3.0:1789760757.739873:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760757.739880:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760757.739896:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24846336 i_blocks 49159 00000020:00200000:3.0:1789760757.739900:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760757.739911:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760757.739945:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24846336, count: 4096 00000080:00200000:3.0:1789760757.739953:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24846336, bytes: 4096 00000080:00200000:3.0:1789760757.739963:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760757.739972:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760757.739974:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24846336, 24850432) 0 0 00000080:00200000:3.0:1789760757.739978:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760757.739994:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6066, 6066] 00020000:00200000:3.0:1789760757.740003:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24846336 chunk: [24846336, 24850432) 24850432, 4096 00020000:00200000:3.0:1789760757.740008:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760757.740026:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6066, 6066] 00000008:00010000:3.0:1789760757.740216:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760757.740222:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760757.740235:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24846336, 24850432) append 0 append lockpos 0 00000080:00200000:3.0:1789760757.740249:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24846336=0x17b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760757.740311:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760757.740313:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24850432 00000020:00010000:3.0:1789760757.740321:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760757.740323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.740325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.740328:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760757.740330:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760757.740336:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760757.740342:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760757.740354:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760757.740539:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 24846336<>24850432 00000100:00100000:2.0:1789760757.740769:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495857920:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760757.769439:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:2.0:1789760757.769454:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760757.769465:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.769627:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6066, 6066] 00000080:00200000:1.0:1789760757.769647:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.769654:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.769670:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24850432 i_blocks 49159 00000020:00200000:1.0:1789760757.769674:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.769684:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.769719:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24850432, count: 4096 00000080:00200000:1.0:1789760757.769726:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24850432, bytes: 4096 00000080:00200000:1.0:1789760757.769735:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.769743:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.769745:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24850432, 24854528) 0 0 00000080:00200000:1.0:1789760757.769750:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.769765:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6067, 6067] 00020000:00200000:1.0:1789760757.769772:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24850432 chunk: [24850432, 24854528) 24854528, 4096 00020000:00200000:1.0:1789760757.769777:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.769797:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6067, 6067] 00000008:00010000:1.0:1789760757.769826:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.769832:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.769843:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24850432, 24854528) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.769856:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24850432=0x17b3000, pages 1 (max 131072), lockless, parallel 00000008:00010000:2.0:1789760757.769942:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:2.0:1789760757.769972:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495857920:192.168.203.109@tcp:4:dd.0 00000020:00200000:1.0:1789760757.770259:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.770263:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24854528 00000020:00010000:1.0:1789760757.770272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.770276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.770279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.770282:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.770285:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.770292:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.770298:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.770312:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760757.770427:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 24850432<>24854528 00000100:00100000:0.0:1789760757.770631:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495858048:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.802961:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973384 gen 2 00000008:00010000:0.0:1789760757.803021:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760757.803029:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.803037:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.803089:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760757.803119:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495858048:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.803147:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6067, 6067] 00000080:00200000:1.0:1789760757.803164:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.803197:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.803213:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24854528 i_blocks 49159 00000020:00200000:1.0:1789760757.803217:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.803226:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.803259:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24854528, count: 4096 00000080:00200000:1.0:1789760757.803266:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24854528, bytes: 4096 00000080:00200000:1.0:1789760757.803275:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.803283:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.803285:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24854528, 24858624) 0 0 00000080:00200000:1.0:1789760757.803289:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.803304:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6068, 6068] 00020000:00200000:1.0:1789760757.803311:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24854528 chunk: [24854528, 24858624) 24858624, 4096 00020000:00200000:1.0:1789760757.803315:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.803332:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6068, 6068] 00000008:00010000:1.0:1789760757.803348:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.803353:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.803362:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24854528, 24858624) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.803376:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24854528=0x17b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.803437:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.803441:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24858624 00000020:00010000:1.0:1789760757.803449:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.803453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.803456:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.803459:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.803625:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.803636:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.803643:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.803661:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760757.803796:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 24854528<>24858624 00000100:00100000:0.0:1789760757.803933:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495858176:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.840009:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973385 gen 2 00000008:00010000:0.0:1789760757.840059:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760757.840074:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.840085:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.840102:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760757.840131:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495858176:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.840137:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6068, 6068] 00000080:00200000:1.0:1789760757.840157:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.840164:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.840222:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24858624 i_blocks 49159 00000020:00200000:1.0:1789760757.840226:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.840237:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.840270:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24858624, count: 4096 00000080:00200000:1.0:1789760757.840276:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24858624, bytes: 4096 00000080:00200000:1.0:1789760757.840286:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.840295:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.840297:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24858624, 24862720) 0 0 00000080:00200000:1.0:1789760757.840301:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.840318:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6069, 6069] 00020000:00200000:1.0:1789760757.840326:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24858624 chunk: [24858624, 24862720) 24862720, 4096 00020000:00200000:1.0:1789760757.840331:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.840351:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6069, 6069] 00000008:00010000:1.0:1789760757.840369:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.840375:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.840387:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24858624, 24862720) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.840401:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24858624=0x17b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.840457:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.840460:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24862720 00000020:00010000:1.0:1789760757.840694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.840699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.840702:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.840705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.840709:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.840717:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.840723:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.840741:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760757.840928:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24858624<>24862720 00000100:00100000:0.0:1789760757.841116:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760757.841136:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495858304:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.885090:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973386 gen 2 00000008:00010000:0.0:1789760757.885143:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760757.885151:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.885162:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.885200:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760757.885242:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495858304:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.885661:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6069, 6069] 00000080:00200000:1.0:1789760757.885678:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.885685:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.885701:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24862720 i_blocks 49159 00000020:00200000:1.0:1789760757.885705:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.885715:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.885749:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24862720, count: 4096 00000080:00200000:1.0:1789760757.885757:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24862720, bytes: 4096 00000080:00200000:1.0:1789760757.885773:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.885783:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.885785:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24862720, 24866816) 0 0 00000080:00200000:1.0:1789760757.885790:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.885808:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6070, 6070] 00020000:00200000:1.0:1789760757.885816:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24862720 chunk: [24862720, 24866816) 24866816, 4096 00020000:00200000:1.0:1789760757.885821:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.885841:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6070, 6070] 00000008:00010000:1.0:1789760757.885860:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.885866:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.885888:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24862720, 24866816) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.885904:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24862720=0x17b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.886078:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.886082:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24866816 00000020:00010000:1.0:1789760757.886091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.886095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.886099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.886102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.886105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.886112:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.886119:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.886251:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760757.886377:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 24862720<>24866816 00000100:00100000:0.0:1789760757.887071:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760757.887095:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495858432:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760757.930508:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760757.930524:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.930536:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.930558:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760757.930587:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495858432:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.930748:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6070, 6070] 00000080:00200000:1.0:1789760757.930764:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.930771:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.930787:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24866816 i_blocks 49159 00000020:00200000:1.0:1789760757.930791:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.930800:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.930842:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24866816, count: 4096 00000080:00200000:1.0:1789760757.930850:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24866816, bytes: 4096 00000080:00200000:1.0:1789760757.930863:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.930872:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.930874:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24866816, 24870912) 0 0 00000080:00200000:1.0:1789760757.930878:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.930896:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6071, 6071] 00020000:00200000:1.0:1789760757.930906:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24866816 chunk: [24866816, 24870912) 24870912, 4096 00020000:00200000:1.0:1789760757.930910:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.930930:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6071, 6071] 00000008:00010000:1.0:1789760757.930948:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.930954:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.930976:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24866816, 24870912) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.930991:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24866816=0x17b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.931046:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.931049:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24870912 00000020:00010000:1.0:1789760757.931058:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.931062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.931064:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.931067:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.931071:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.931078:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.931084:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.931100:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760757.931264:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 24866816<>24870912 00000100:00100000:0.0:1789760757.931752:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760757.931773:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495858560:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760757.987020:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973387 gen 2 00000100:00100000:0.0:1789760757.987062:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504e300 x1876699495858560/t4294973388(4294973388) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760773 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760757.987111:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760757.987119:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.987147:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.987222:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760757.987249:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495858560:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760757.987477:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6071, 6071] 00000080:00200000:1.0:1789760757.987506:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.987513:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.987529:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24870912 i_blocks 49159 00000020:00200000:1.0:1789760757.987532:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.987542:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.987580:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24870912, count: 4096 00000080:00200000:1.0:1789760757.987588:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24870912, bytes: 4096 00000080:00200000:1.0:1789760757.987603:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.987612:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.987615:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24870912, 24875008) 0 0 00000080:00200000:1.0:1789760757.987620:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.987640:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6072, 6072] 00020000:00200000:1.0:1789760757.987648:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24870912 chunk: [24870912, 24875008) 24875008, 4096 00020000:00200000:1.0:1789760757.987662:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.987685:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6072, 6072] 00000008:00010000:1.0:1789760757.987704:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.987709:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760757.987722:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24870912, 24875008) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.987736:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24870912=0x17b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.987797:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.987800:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24875008 00000020:00010000:1.0:1789760757.987809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.987812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.987815:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.987818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.987823:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.987830:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.987836:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.987855:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760757.987985:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 24870912<>24875008 00000100:00100000:0.0:1789760757.988828:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760757.988854:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495858688:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760758.041388:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973389 gen 2 00000008:00010000:0.0:1789760758.041455:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760758.041622:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.041633:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.041780:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760758.041812:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495858688:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760758.041903:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6072, 6072] 00000080:00200000:1.0:1789760758.041922:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.041931:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.041950:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24875008 i_blocks 49159 00000020:00200000:1.0:1789760758.041953:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.041963:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.042000:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24875008, count: 4096 00000080:00200000:1.0:1789760758.042008:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24875008, bytes: 4096 00000080:00200000:1.0:1789760758.042021:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.042031:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.042034:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24875008, 24879104) 0 0 00000080:00200000:1.0:1789760758.042038:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.042058:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6073, 6073] 00020000:00200000:1.0:1789760758.042067:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24875008 chunk: [24875008, 24879104) 24879104, 4096 00020000:00200000:1.0:1789760758.042071:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.042091:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6073, 6073] 00000008:00010000:1.0:1789760758.042108:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.042114:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760758.042125:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24875008, 24879104) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.042234:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24875008=0x17b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.042293:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.042296:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24879104 00000020:00010000:1.0:1789760758.042306:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.042309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.042313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.042316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.042320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.042329:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.042335:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.042353:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760758.042626:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 24875008<>24879104 00000100:00100000:0.0:1789760758.042757:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495858816:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760758.090947:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973390 gen 2 00000008:00010000:0.0:1789760758.091344:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760758.091353:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.091365:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.091447:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760758.091479:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495858816:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760758.091566:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6073, 6073] 00000080:00200000:2.0:1789760758.091583:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760758.091589:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760758.091604:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24879104 i_blocks 49159 00000020:00200000:2.0:1789760758.091609:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760758.091619:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760758.091658:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24879104, count: 4096 00000080:00200000:2.0:1789760758.091665:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24879104, bytes: 4096 00000080:00200000:2.0:1789760758.091677:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760758.091686:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760758.091689:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24879104, 24883200) 0 0 00000080:00200000:2.0:1789760758.091693:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760758.091712:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6074, 6074] 00020000:00200000:2.0:1789760758.091722:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24879104 chunk: [24879104, 24883200) 24883200, 4096 00020000:00200000:2.0:1789760758.091726:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760758.091750:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6074, 6074] 00000008:00010000:2.0:1789760758.091836:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760758.091842:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760758.091921:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24879104, 24883200) append 0 append lockpos 0 00000080:00200000:2.0:1789760758.091936:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24879104=0x17ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760758.091997:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760758.092000:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24883200 00000020:00010000:2.0:1789760758.092009:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760758.092013:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.092016:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.092019:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760758.092023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760758.092030:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760758.092107:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760758.092126:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760758.092372:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 24879104<>24883200 00000100:00100000:2.0:1789760758.092415:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760758.092433:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495858944:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.133400:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973391 gen 2 00000008:00010000:2.0:1789760758.133559:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760758.133570:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.133583:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.133608:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760758.133640:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495858944:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.134104:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6074, 6074] 00000080:00200000:3.0:1789760758.134124:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.134131:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.134359:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24883200 i_blocks 49159 00000020:00200000:3.0:1789760758.134364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.134714:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.134754:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24883200, count: 4096 00000080:00200000:3.0:1789760758.134929:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24883200, bytes: 4096 00000080:00200000:3.0:1789760758.134942:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.134952:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.134954:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24883200, 24887296) 0 0 00000080:00200000:3.0:1789760758.134960:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.135043:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6075, 6075] 00020000:00200000:3.0:1789760758.135051:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24883200 chunk: [24883200, 24887296) 24887296, 4096 00020000:00200000:3.0:1789760758.135056:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.135080:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6075, 6075] 00000008:00010000:3.0:1789760758.135099:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.135104:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.135117:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24883200, 24887296) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.135132:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24883200=0x17bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.135234:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.135237:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24887296 00000020:00010000:3.0:1789760758.135246:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.135249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.135253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.135256:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.135259:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.135267:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.135274:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.135299:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760758.135424:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 24883200<>24887296 00000100:00100000:2.0:1789760758.135633:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859072:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760758.187934:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760758.187953:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.188433:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.188468:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760758.188509:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859072:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760758.188630:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6075, 6075] 00000080:00200000:0.0:1789760758.188648:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.188655:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.188673:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24887296 i_blocks 49159 00000020:00200000:0.0:1789760758.188677:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.188687:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.188726:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24887296, count: 4096 00000080:00200000:0.0:1789760758.188733:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24887296, bytes: 4096 00000080:00200000:0.0:1789760758.188746:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.188756:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.188759:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24887296, 24891392) 0 0 00000080:00200000:0.0:1789760758.188763:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.188780:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6076, 6076] 00020000:00200000:0.0:1789760758.188789:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24887296 chunk: [24887296, 24891392) 24891392, 4096 00020000:00200000:0.0:1789760758.188794:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.188940:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6076, 6076] 00000008:00010000:0.0:1789760758.188960:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.188966:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760758.188979:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24887296, 24891392) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.188995:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24887296=0x17bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.189058:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.189061:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24891392 00000020:00010000:0.0:1789760758.189069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.189073:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.189077:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.189081:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.189084:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.189093:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.189099:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.189117:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760758.189387:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 24887296<>24891392 00000100:00100000:1.0:1789760758.189453:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495859200:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760758.219975:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760758.221492:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973392 gen 2 00000100:00100000:0.0:1789760758.221530:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211835f80 x1876699495859200/t4294973393(4294973393) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760774 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760758.221578:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760758.221585:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.221594:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.221616:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760758.221639:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495859200:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760758.221725:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6076, 6076] 00000080:00200000:1.0:1789760758.221745:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.221750:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.221766:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24891392 i_blocks 49159 00000020:00200000:1.0:1789760758.221769:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.221778:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.221814:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24891392, count: 4096 00000080:00200000:1.0:1789760758.221822:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24891392, bytes: 4096 00000080:00200000:1.0:1789760758.221832:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.221842:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.221844:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24891392, 24895488) 0 0 00000080:00200000:1.0:1789760758.221849:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.221867:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6077, 6077] 00020000:00200000:1.0:1789760758.221876:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24891392 chunk: [24891392, 24895488) 24895488, 4096 00020000:00200000:1.0:1789760758.221881:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.221904:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6077, 6077] 00000008:00010000:1.0:1789760758.221926:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.221934:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760758.221946:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24891392, 24895488) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.221960:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24891392=0x17bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.222021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.222024:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24895488 00000020:00010000:1.0:1789760758.222035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.222038:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.222051:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.222055:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.222058:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.222066:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.222073:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.222088:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760758.222239:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 24891392<>24895488 00000100:00100000:0.0:1789760758.222555:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495859328:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760758.264125:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973393 gen 2 00000100:00100000:0.0:1789760758.264204:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d180 x1876699495859328/t4294973394(4294973394) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760774 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760758.264262:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760758.264269:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.264279:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.264343:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760758.264367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495859328:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760758.265709:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6077, 6077] 00000080:00200000:1.0:1789760758.265733:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.265741:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.265832:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24895488 i_blocks 49159 00000020:00200000:1.0:1789760758.265838:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.265851:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.265883:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24895488, count: 4096 00000080:00200000:1.0:1789760758.265891:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24895488, bytes: 4096 00000080:00200000:1.0:1789760758.265903:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.265912:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.265915:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24895488, 24899584) 0 0 00000080:00200000:1.0:1789760758.265919:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.265938:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6078, 6078] 00020000:00200000:1.0:1789760758.265947:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24895488 chunk: [24895488, 24899584) 24899584, 4096 00020000:00200000:1.0:1789760758.265952:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.265975:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6078, 6078] 00000008:00010000:1.0:1789760758.265996:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.266002:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760758.266015:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24895488, 24899584) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.266029:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24895488=0x17be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.266093:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.266097:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24899584 00000020:00010000:1.0:1789760758.266106:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.266109:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.266113:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.266116:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.266119:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.266127:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.266135:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.266153:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760758.268655:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 24895488<>24899584 00000100:00100000:1.0:1789760758.268746:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760758.268937:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495859456:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760758.324710:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973394 gen 2 00000100:00100000:0.0:1789760758.324753:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d500 x1876699495859456/t4294973395(4294973395) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760774 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760758.324800:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760758.324808:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.324818:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.324844:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760758.324880:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495859456:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760758.325004:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6078, 6078] 00000080:00200000:2.0:1789760758.325025:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760758.325031:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760758.325047:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24899584 i_blocks 49159 00000020:00200000:2.0:1789760758.325051:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760758.325061:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760758.325098:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24899584, count: 4096 00000080:00200000:2.0:1789760758.325106:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24899584, bytes: 4096 00000080:00200000:2.0:1789760758.325117:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760758.325127:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760758.325130:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24899584, 24903680) 0 0 00000080:00200000:2.0:1789760758.325135:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760758.325154:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6079, 6079] 00020000:00200000:2.0:1789760758.325163:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24899584 chunk: [24899584, 24903680) 24903680, 4096 00020000:00200000:2.0:1789760758.325200:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760758.325224:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6079, 6079] 00000008:00010000:2.0:1789760758.325244:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760758.325251:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760758.325263:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24899584, 24903680) append 0 append lockpos 0 00000080:00200000:2.0:1789760758.325280:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24899584=0x17bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760758.325340:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760758.325344:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24903680 00000020:00010000:2.0:1789760758.325353:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760758.325357:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.325360:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.325364:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760758.325367:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760758.325376:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760758.325382:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760758.325400:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760758.325544:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 24899584<>24903680 00000100:00100000:2.0:1789760758.325593:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760758.325613:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859584:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.360592:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973395 gen 2 00000100:00100000:2.0:1789760758.360635:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265f80 x1876699495859584/t4294973396(4294973396) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760774 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760758.360685:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760758.360692:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.360703:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.360768:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760758.360791:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859584:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.360975:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6079, 6079] 00000080:00200000:3.0:1789760758.360993:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.361000:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.361015:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24903680 i_blocks 49159 00000020:00200000:3.0:1789760758.361019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.361028:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.361134:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24903680, count: 4096 00000080:00200000:3.0:1789760758.361143:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24903680, bytes: 4096 00000080:00200000:3.0:1789760758.361153:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.361162:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.361165:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24903680, 24907776) 0 0 00000080:00200000:3.0:1789760758.361202:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.361221:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6080, 6080] 00020000:00200000:3.0:1789760758.361230:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24903680 chunk: [24903680, 24907776) 24907776, 4096 00020000:00200000:3.0:1789760758.361235:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.361255:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6080, 6080] 00000008:00010000:3.0:1789760758.361275:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.361281:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.361293:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24903680, 24907776) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.361308:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24903680=0x17c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.361366:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.361369:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24907776 00000020:00010000:3.0:1789760758.361377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.361381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.361384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.361388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.361391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.361398:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.361405:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.361422:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760758.361615:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 24903680<>24907776 00000100:00100000:2.0:1789760758.361678:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859712:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.401897:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973397 gen 2 00000008:00010000:2.0:1789760758.401966:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:2.0:1789760758.401977:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.402200:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.402311:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:2.0:1789760758.402343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859712:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.412409:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6080, 6080] 00000080:00200000:3.0:1789760758.412441:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.412449:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.412471:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24907776 i_blocks 49159 00000020:00200000:3.0:1789760758.412477:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.412504:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.412541:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24907776, count: 4096 00000080:00200000:3.0:1789760758.412549:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24907776, bytes: 4096 00000080:00200000:3.0:1789760758.412564:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.412575:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.412578:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24907776, 24911872) 0 0 00000080:00200000:3.0:1789760758.412582:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.412599:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6081, 6081] 00020000:00200000:3.0:1789760758.412608:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24907776 chunk: [24907776, 24911872) 24911872, 4096 00020000:00200000:3.0:1789760758.412613:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.412636:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6081, 6081] 00000008:00010000:3.0:1789760758.412656:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.412662:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.412675:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24907776, 24911872) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.412693:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24907776=0x17c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.412782:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.412786:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24911872 00000020:00010000:3.0:1789760758.412795:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.412799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.412803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.412807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.412810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.412818:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.412826:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.412861:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760758.413023:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 24907776<>24911872 00000100:00100000:2.0:1789760758.413308:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760758.413336:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859840:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.456661:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973398 gen 2 00000008:00010000:2.0:1789760758.456711:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:2.0:1789760758.456719:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.456731:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.456753:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:2.0:1789760758.456784:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859840:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.456988:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6081, 6081] 00000080:00200000:3.0:1789760758.457005:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.457011:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.457028:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24911872 i_blocks 49159 00000020:00200000:3.0:1789760758.457032:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.457041:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.457076:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24911872, count: 4096 00000080:00200000:3.0:1789760758.457084:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24911872, bytes: 4096 00000080:00200000:3.0:1789760758.457093:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.457103:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.457106:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24911872, 24915968) 0 0 00000080:00200000:3.0:1789760758.457111:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.457129:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6082, 6082] 00020000:00200000:3.0:1789760758.457148:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24911872 chunk: [24911872, 24915968) 24915968, 4096 00020000:00200000:3.0:1789760758.457153:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.457213:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6082, 6082] 00000008:00010000:3.0:1789760758.457233:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.457239:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.457251:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24911872, 24915968) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.457266:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24911872=0x17c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.457321:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.457324:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24915968 00000020:00010000:3.0:1789760758.457333:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.457337:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.457341:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.457345:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.457348:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.457357:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.457364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.457381:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760758.457513:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 24911872<>24915968 00000100:00100000:2.0:1789760758.457683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859968:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.488961:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973399 gen 2 00000008:00010000:2.0:1789760758.489007:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:2.0:1789760758.489015:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.489025:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.489091:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:2.0:1789760758.489118:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495859968:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.489588:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6082, 6082] 00000080:00200000:3.0:1789760758.489604:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.489610:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.489626:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24915968 i_blocks 49159 00000020:00200000:3.0:1789760758.489630:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.489639:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.489671:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24915968, count: 4096 00000080:00200000:3.0:1789760758.489678:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24915968, bytes: 4096 00000080:00200000:3.0:1789760758.489690:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.489699:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.489702:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24915968, 24920064) 0 0 00000080:00200000:3.0:1789760758.489706:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.489723:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6083, 6083] 00020000:00200000:3.0:1789760758.489731:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24915968 chunk: [24915968, 24920064) 24920064, 4096 00020000:00200000:3.0:1789760758.489735:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.489755:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6083, 6083] 00000008:00010000:3.0:1789760758.489772:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.489778:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.489788:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24915968, 24920064) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.489913:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24915968=0x17c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.489966:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.489968:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24920064 00000020:00010000:3.0:1789760758.489977:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.489980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.489983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.489987:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.489990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.489998:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.490004:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.490019:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760758.490133:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 24915968<>24920064 00000100:00100000:2.0:1789760758.490268:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495860096:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.514809:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973400 gen 2 00000008:00010000:2.0:1789760758.514924:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:2.0:1789760758.514933:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.514943:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.514966:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:2.0:1789760758.514996:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495860096:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.515289:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6083, 6083] 00000080:00200000:3.0:1789760758.515306:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.515312:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.515328:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24920064 i_blocks 49159 00000020:00200000:3.0:1789760758.515331:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.515341:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.515374:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24920064, count: 4096 00000080:00200000:3.0:1789760758.515534:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24920064, bytes: 4096 00000080:00200000:3.0:1789760758.515546:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.515555:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.515557:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24920064, 24924160) 0 0 00000080:00200000:3.0:1789760758.515562:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.515577:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6084, 6084] 00020000:00200000:3.0:1789760758.515585:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24920064 chunk: [24920064, 24924160) 24924160, 4096 00020000:00200000:3.0:1789760758.515590:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.515608:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6084, 6084] 00000008:00010000:3.0:1789760758.515623:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.515628:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.515638:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24920064, 24924160) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.515651:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24920064=0x17c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.515700:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.515703:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24924160 00000020:00010000:3.0:1789760758.515711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.515714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.515717:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.515720:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.515723:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.515730:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.515736:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.515751:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760758.515863:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 24920064<>24924160 00000100:00100000:2.0:1789760758.515999:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495860224:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.547784:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973401 gen 2 00000008:00010000:2.0:1789760758.547838:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:2.0:1789760758.547848:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.547859:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.547886:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:2.0:1789760758.547916:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495860224:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.548849:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6084, 6084] 00000080:00200000:3.0:1789760758.548869:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.548875:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.548892:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24924160 i_blocks 49159 00000020:00200000:3.0:1789760758.548896:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.548906:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.548940:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24924160, count: 4096 00000080:00200000:3.0:1789760758.548947:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24924160, bytes: 4096 00000080:00200000:3.0:1789760758.548959:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.548975:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.548977:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24924160, 24928256) 0 0 00000080:00200000:3.0:1789760758.548982:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.548998:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6085, 6085] 00020000:00200000:3.0:1789760758.549006:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24924160 chunk: [24924160, 24928256) 24928256, 4096 00020000:00200000:3.0:1789760758.549011:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.549031:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6085, 6085] 00000008:00010000:3.0:1789760758.549047:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.549052:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.549063:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24924160, 24928256) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.549078:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24924160=0x17c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.549137:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.549141:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24928256 00000020:00010000:3.0:1789760758.549150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.549249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.549253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.549257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.549260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.549270:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.549277:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.549294:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760758.549427:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 24924160<>24928256 00000100:00100000:2.0:1789760758.549686:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760758.549710:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495860352:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.583811:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973402 gen 2 00000008:00010000:2.0:1789760758.583862:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:2.0:1789760758.583869:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.583881:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.583905:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:2.0:1789760758.583935:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495860352:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.584086:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6085, 6085] 00000080:00200000:3.0:1789760758.584103:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.584110:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.584124:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24928256 i_blocks 49159 00000020:00200000:3.0:1789760758.584128:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.584137:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.584206:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24928256, count: 4096 00000080:00200000:3.0:1789760758.584213:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24928256, bytes: 4096 00000080:00200000:3.0:1789760758.584222:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.584232:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.584235:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24928256, 24932352) 0 0 00000080:00200000:3.0:1789760758.584239:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.584257:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6086, 6086] 00020000:00200000:3.0:1789760758.584266:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24928256 chunk: [24928256, 24932352) 24932352, 4096 00020000:00200000:3.0:1789760758.584271:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.584289:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6086, 6086] 00000008:00010000:3.0:1789760758.584306:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.584311:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.584321:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24928256, 24932352) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.584340:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24928256=0x17c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.584393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.584396:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24932352 00000020:00010000:3.0:1789760758.584405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.584408:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.584411:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.584415:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.584418:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.584426:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.584433:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.584449:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760758.584580:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 24928256<>24932352 00000100:00100000:2.0:1789760758.584718:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495860480:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.620221:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973403 gen 2 00000008:00010000:2.0:1789760758.620274:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:2.0:1789760758.620283:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.620294:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.620592:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:2.0:1789760758.620619:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495860480:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.620929:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6086, 6086] 00000080:00200000:3.0:1789760758.620947:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.620954:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.620971:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24932352 i_blocks 49159 00000020:00200000:3.0:1789760758.620974:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.620982:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.621012:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24932352, count: 4096 00000080:00200000:3.0:1789760758.621018:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24932352, bytes: 4096 00000080:00200000:3.0:1789760758.621028:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.621036:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.621039:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24932352, 24936448) 0 0 00000080:00200000:3.0:1789760758.621043:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.621058:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6087, 6087] 00020000:00200000:3.0:1789760758.621066:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24932352 chunk: [24932352, 24936448) 24936448, 4096 00020000:00200000:3.0:1789760758.621069:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.621084:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6087, 6087] 00000008:00010000:3.0:1789760758.621098:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.621102:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.621111:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24932352, 24936448) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.621220:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24932352=0x17c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.621280:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.621283:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24936448 00000020:00010000:3.0:1789760758.621292:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.621296:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.621300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.621303:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.621306:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.621313:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.621321:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.621337:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760758.621523:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 24932352<>24936448 00000100:00100000:2.0:1789760758.621580:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495860608:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.665094:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973404 gen 2 00000008:00010000:2.0:1789760758.666688:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:2.0:1789760758.666702:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.666715:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.666822:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:2.0:1789760758.666927:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495860608:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.667026:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6087, 6087] 00000080:00200000:3.0:1789760758.667043:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.667049:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.667064:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24936448 i_blocks 49159 00000020:00200000:3.0:1789760758.667068:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.667077:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.667111:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24936448, count: 4096 00000080:00200000:3.0:1789760758.667119:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24936448, bytes: 4096 00000080:00200000:3.0:1789760758.667133:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.667142:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.667144:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24936448, 24940544) 0 0 00000080:00200000:3.0:1789760758.667149:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.667166:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6088, 6088] 00020000:00200000:3.0:1789760758.667775:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24936448 chunk: [24936448, 24940544) 24940544, 4096 00020000:00200000:3.0:1789760758.667781:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.667804:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6088, 6088] 00000008:00010000:3.0:1789760758.667835:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.667841:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.667854:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24936448, 24940544) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.667870:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24936448=0x17c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.667929:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.667932:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24940544 00000020:00010000:3.0:1789760758.667940:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.667943:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.667946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.667950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.667953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.667961:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.667967:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.667986:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760758.668111:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 24936448<>24940544 00000100:00100000:3.0:1789760758.668182:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495860736:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760758.669816:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495860864:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760758.670359:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760758.670369:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495860992:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760758.677316:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760758.677352:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495861120:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760758.677686:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495860992:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760758.677706:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495860864:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760758.679234:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495861120:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760758.723030:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973405 gen 2 00000008:00010000:3.0:1789760758.723078:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:3.0:1789760758.723087:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.723099:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.723126:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:3.0:1789760758.723157:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495860736:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760758.723328:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6088, 6088] 00000080:00200000:0.0:1789760758.723345:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.723358:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.723374:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24940544 i_blocks 49159 00000020:00200000:0.0:1789760758.723377:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.723386:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.723421:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24940544, count: 4096 00000080:00200000:0.0:1789760758.723428:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24940544, bytes: 4096 00000080:00200000:0.0:1789760758.723438:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.723447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.723451:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24940544, 24944640) 0 0 00000080:00200000:0.0:1789760758.723455:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.723472:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6089, 6089] 00020000:00200000:0.0:1789760758.723491:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24940544 chunk: [24940544, 24944640) 24944640, 4096 00020000:00200000:0.0:1789760758.723496:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.723516:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6089, 6089] 00000008:00010000:0.0:1789760758.723533:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.723538:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760758.723550:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24940544, 24944640) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.723564:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24940544=0x17c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.723620:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.723623:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24944640 00000020:00010000:0.0:1789760758.723645:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.723649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.723653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.723656:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.723659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.723666:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.723672:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.723688:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760758.723821:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 24940544<>24944640 00000100:00100000:1.0:1789760758.724495:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495861248:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760758.753713:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973406 gen 2 00000008:00010000:1.0:1789760758.753773:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760758.753782:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.753793:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.753815:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760758.753846:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495861248:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760758.753889:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6089, 6089] 00000080:00200000:0.0:1789760758.753908:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.753915:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.754017:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24944640 i_blocks 49159 00000020:00200000:1.0:1789760758.754022:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.754033:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.754070:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24944640, count: 4096 00000080:00200000:1.0:1789760758.754078:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24944640, bytes: 4096 00000080:00200000:1.0:1789760758.754088:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.754098:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.754101:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24944640, 24948736) 0 0 00000080:00200000:1.0:1789760758.754106:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.754124:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6090, 6090] 00020000:00200000:1.0:1789760758.754133:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24944640 chunk: [24944640, 24948736) 24948736, 4096 00020000:00200000:1.0:1789760758.754138:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.754160:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6090, 6090] 00000008:00010000:1.0:1789760758.754212:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.754219:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760758.754231:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24944640, 24948736) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.754245:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24944640=0x17ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.754300:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.754303:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24948736 00000020:00010000:1.0:1789760758.754312:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.754315:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.754318:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.754322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.754325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.754333:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.754339:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.754354:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760758.754498:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 24944640<>24948736 00000100:00100000:1.0:1789760758.754544:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760758.754560:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495861376:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760758.782251:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760758.782266:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.782278:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.782302:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760758.782333:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495861376:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.782428:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6090, 6090] 00000080:00200000:3.0:1789760758.782446:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.782453:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.782468:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24948736 i_blocks 49159 00000020:00200000:3.0:1789760758.782472:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.782491:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.782530:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24948736, count: 4096 00000080:00200000:3.0:1789760758.782537:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24948736, bytes: 4096 00000080:00200000:3.0:1789760758.782548:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.782557:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.782560:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24948736, 24952832) 0 0 00000080:00200000:3.0:1789760758.782565:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.782582:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6091, 6091] 00020000:00200000:3.0:1789760758.782590:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24948736 chunk: [24948736, 24952832) 24952832, 4096 00020000:00200000:3.0:1789760758.782595:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.782615:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6091, 6091] 00000008:00010000:3.0:1789760758.782634:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.782640:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.782651:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24948736, 24952832) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.782668:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24948736=0x17cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.782729:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.782732:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24952832 00000020:00010000:3.0:1789760758.782742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.782747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.782750:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.782754:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.782757:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.782767:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.782774:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.782793:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760758.782934:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 24948736<>24952832 00000100:00100000:3.0:1789760758.782986:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760758.783004:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495861504:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760758.835023:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973407 gen 2 00000100:00100000:3.0:1789760758.835067:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203144000 x1876699495861504/t4294973408(4294973408) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760774 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760758.835115:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:3.0:1789760758.835122:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.835131:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.835158:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:3.0:1789760758.835214:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495861504:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760758.835366:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6091, 6091] 00000080:00200000:1.0:1789760758.835383:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.835389:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.835405:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24952832 i_blocks 49159 00000020:00200000:1.0:1789760758.835409:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.835418:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.835453:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24952832, count: 4096 00000080:00200000:1.0:1789760758.835461:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24952832, bytes: 4096 00000080:00200000:1.0:1789760758.835470:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.835491:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.835494:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24952832, 24956928) 0 0 00000080:00200000:1.0:1789760758.835498:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.835515:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6092, 6092] 00020000:00200000:1.0:1789760758.835523:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24952832 chunk: [24952832, 24956928) 24956928, 4096 00020000:00200000:1.0:1789760758.835528:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.835550:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6092, 6092] 00000008:00010000:1.0:1789760758.835568:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.835574:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760758.835587:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24952832, 24956928) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.835601:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24952832=0x17cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.835663:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.835666:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24956928 00000020:00010000:1.0:1789760758.835674:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.835678:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.835681:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.835684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.835688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.835695:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.835702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.835719:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760758.836384:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 24952832<>24956928 00000100:00100000:0.0:1789760758.836441:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760758.836558:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495861632:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760758.866155:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973409 gen 2 00000008:00010000:1.0:1789760758.866252:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760758.866260:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.866270:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.866290:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760758.866319:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495861632:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760758.866895:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6092, 6092] 00000080:00200000:0.0:1789760758.866914:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.866920:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.866937:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24956928 i_blocks 49159 00000020:00200000:0.0:1789760758.866941:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.866949:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.866981:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24956928, count: 4096 00000080:00200000:0.0:1789760758.866986:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24956928, bytes: 4096 00000080:00200000:0.0:1789760758.866995:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.867001:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.867003:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24956928, 24961024) 0 0 00000080:00200000:0.0:1789760758.867007:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.867020:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6093, 6093] 00020000:00200000:0.0:1789760758.867027:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24956928 chunk: [24956928, 24961024) 24961024, 4096 00020000:00200000:0.0:1789760758.867030:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.867045:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6093, 6093] 00000008:00010000:0.0:1789760758.867058:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.867062:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760758.867071:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24956928, 24961024) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.867083:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24956928=0x17cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.867130:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.867132:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24961024 00000020:00010000:0.0:1789760758.867139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.867141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.867144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.867146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.867148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.867153:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.867158:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.867199:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760758.867325:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 24956928<>24961024 00000100:00100000:1.0:1789760758.867519:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495861760:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760758.895604:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973410 gen 2 00000008:00010000:1.0:1789760758.895657:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760758.895666:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.895677:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.895750:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760758.895781:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495861760:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.895922:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6093, 6093] 00000080:00200000:3.0:1789760758.895939:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.895946:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.895961:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24961024 i_blocks 49159 00000020:00200000:3.0:1789760758.895965:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.895974:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.896010:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24961024, count: 4096 00000080:00200000:3.0:1789760758.896017:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24961024, bytes: 4096 00000080:00200000:3.0:1789760758.896027:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.896036:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.896038:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24961024, 24965120) 0 0 00000080:00200000:3.0:1789760758.896043:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.896059:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6094, 6094] 00020000:00200000:3.0:1789760758.896068:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24961024 chunk: [24961024, 24965120) 24965120, 4096 00020000:00200000:3.0:1789760758.896072:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.896093:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6094, 6094] 00000008:00010000:3.0:1789760758.896112:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.896118:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.896129:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24961024, 24965120) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.896143:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24961024=0x17ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.896218:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.896221:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24965120 00000020:00010000:3.0:1789760758.896230:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.896234:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.896237:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.896240:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.896243:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.896251:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.896259:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.896276:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760758.896393:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 24961024<>24965120 00000100:00100000:2.0:1789760758.896484:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495861888:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.944961:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973411 gen 2 00000008:00010000:2.0:1789760758.945022:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:2.0:1789760758.945031:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.945111:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.945135:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:2.0:1789760758.945166:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495861888:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.945325:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6094, 6094] 00000080:00200000:3.0:1789760758.945347:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.945354:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.945372:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24965120 i_blocks 49159 00000020:00200000:3.0:1789760758.945375:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.945386:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.945423:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24965120, count: 4096 00000080:00200000:3.0:1789760758.945432:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24965120, bytes: 4096 00000080:00200000:3.0:1789760758.945443:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.945452:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.945454:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24965120, 24969216) 0 0 00000080:00200000:3.0:1789760758.945459:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.945485:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6095, 6095] 00020000:00200000:3.0:1789760758.945495:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24965120 chunk: [24965120, 24969216) 24969216, 4096 00020000:00200000:3.0:1789760758.945500:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.945522:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6095, 6095] 00000008:00010000:3.0:1789760758.946302:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.946313:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.946335:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24965120, 24969216) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.946354:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24965120=0x17cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.946431:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.946435:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24969216 00000020:00010000:3.0:1789760758.946445:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.946449:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.946453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.946468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.946473:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.946492:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.946501:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.946519:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760758.946800:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 24965120<>24969216 00000100:00100000:2.0:1789760758.946923:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495862016:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760758.996434:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973412 gen 2 00000008:00010000:2.0:1789760758.996506:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:2.0:1789760758.996515:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.996526:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.996549:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:2.0:1789760758.996584:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495862016:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760758.996707:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6095, 6095] 00000080:00200000:3.0:1789760758.996724:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.996731:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.996748:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24969216 i_blocks 49159 00000020:00200000:3.0:1789760758.996752:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.996764:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.996798:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24969216, count: 4096 00000080:00200000:3.0:1789760758.996806:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24969216, bytes: 4096 00000080:00200000:3.0:1789760758.996816:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.996825:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.996827:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24969216, 24973312) 0 0 00000080:00200000:3.0:1789760758.996832:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.996849:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6096, 6096] 00020000:00200000:3.0:1789760758.996857:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24969216 chunk: [24969216, 24973312) 24973312, 4096 00020000:00200000:3.0:1789760758.996862:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.996882:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6096, 6096] 00000008:00010000:3.0:1789760758.996899:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.996904:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760758.996916:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24969216, 24973312) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.996932:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24969216=0x17d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.997003:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.997006:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24973312 00000020:00010000:3.0:1789760758.997015:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.997018:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.997022:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.997025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.997028:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.997036:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.997043:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.997060:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760758.997440:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 24969216<>24973312 00000100:00100000:3.0:1789760758.997500:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760758.997516:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495862144:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760759.027257:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760759.027274:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.027285:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.027380:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760759.027409:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495862144:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760759.027549:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6096, 6096] 00000080:00200000:2.0:1789760759.027566:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.027572:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.027588:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24973312 i_blocks 49159 00000020:00200000:2.0:1789760759.027592:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.027601:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.027636:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24973312, count: 4096 00000080:00200000:2.0:1789760759.027644:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24973312, bytes: 4096 00000080:00200000:2.0:1789760759.027656:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.027664:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.027666:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24973312, 24977408) 0 0 00000080:00200000:2.0:1789760759.027671:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.027688:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6097, 6097] 00020000:00200000:2.0:1789760759.027697:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24973312 chunk: [24973312, 24977408) 24977408, 4096 00020000:00200000:2.0:1789760759.027702:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.027724:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6097, 6097] 00000008:00010000:2.0:1789760759.027743:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.027749:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760759.027761:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24973312, 24977408) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.027777:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24973312=0x17d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.027960:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.027963:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24977408 00000020:00010000:2.0:1789760759.027972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.027975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.027979:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.027982:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.027985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.027993:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.028000:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.028017:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760759.028146:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 24973312<>24977408 00000100:00100000:3.0:1789760759.028222:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495862272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760759.057604:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973414 gen 2 00000008:00010000:3.0:1789760759.057650:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760759.057656:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.057665:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.057687:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760759.057712:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495862272:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.057799:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6097, 6097] 00000080:00200000:0.0:1789760759.057813:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.057818:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.057841:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24977408 i_blocks 49159 00000020:00200000:0.0:1789760759.057844:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.057851:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.057874:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24977408, count: 4096 00000080:00200000:0.0:1789760759.057879:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24977408, bytes: 4096 00000080:00200000:0.0:1789760759.057886:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.057893:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.057895:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24977408, 24981504) 0 0 00000080:00200000:0.0:1789760759.057898:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.057910:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6098, 6098] 00020000:00200000:0.0:1789760759.057917:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24977408 chunk: [24977408, 24981504) 24981504, 4096 00020000:00200000:0.0:1789760759.057924:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.057940:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6098, 6098] 00000008:00010000:0.0:1789760759.057953:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.057958:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.057967:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24977408, 24981504) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.057978:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24977408=0x17d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.058022:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.058024:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24981504 00000020:00010000:0.0:1789760759.058087:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.058090:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.058093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.058096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.058100:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.058106:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.058111:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.058126:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760759.058283:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 24977408<>24981504 00000100:00100000:1.0:1789760759.058512:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495862400:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760759.120306:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973415 gen 2 00000008:00010000:1.0:1789760759.120357:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760759.120365:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.120377:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.120398:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760759.120426:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495862400:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.120701:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6098, 6098] 00000080:00200000:0.0:1789760759.120720:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.120726:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.120745:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24981504 i_blocks 49159 00000020:00200000:0.0:1789760759.120749:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.120759:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.120794:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24981504, count: 4096 00000080:00200000:0.0:1789760759.120802:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24981504, bytes: 4096 00000080:00200000:0.0:1789760759.120814:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.120823:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.120825:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24981504, 24985600) 0 0 00000080:00200000:0.0:1789760759.120830:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.120846:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6099, 6099] 00020000:00200000:0.0:1789760759.120866:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24981504 chunk: [24981504, 24985600) 24985600, 4096 00020000:00200000:0.0:1789760759.120871:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.120893:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6099, 6099] 00000008:00010000:0.0:1789760759.120910:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.120917:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.120928:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24981504, 24985600) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.120944:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24981504=0x17d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.121018:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.121021:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24985600 00000020:00010000:0.0:1789760759.121227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.121232:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.121236:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.121239:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.121242:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.121250:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.121258:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.121277:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760759.121404:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 24981504<>24985600 00000100:00100000:1.0:1789760759.121559:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760759.121578:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495862528:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760759.177865:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973416 gen 2 00000008:00010000:1.0:1789760759.177919:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760759.177928:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.177941:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.177960:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760759.177988:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495862528:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.178211:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6099, 6099] 00000080:00200000:0.0:1789760759.178229:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.178236:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.178253:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24985600 i_blocks 49159 00000020:00200000:0.0:1789760759.178257:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.178267:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.178300:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24985600, count: 4096 00000080:00200000:0.0:1789760759.178307:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24985600, bytes: 4096 00000080:00200000:0.0:1789760759.178317:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.178327:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.178329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24985600, 24989696) 0 0 00000080:00200000:0.0:1789760759.178333:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.178349:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6100, 6100] 00020000:00200000:0.0:1789760759.178358:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24985600 chunk: [24985600, 24989696) 24989696, 4096 00020000:00200000:0.0:1789760759.178362:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.178569:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6100, 6100] 00000008:00010000:0.0:1789760759.178590:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.178595:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.178607:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24985600, 24989696) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.178622:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24985600=0x17d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.179564:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.179571:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24989696 00000020:00010000:0.0:1789760759.179583:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.179587:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.179590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.179593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.179596:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.179606:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.179615:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.179640:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760759.180133:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 24985600<>24989696 00000100:00100000:0.0:1789760759.180473:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495862656:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760759.221052:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973417 gen 2 00000008:00010000:1.0:1789760759.221108:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760759.221117:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.221127:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.221243:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760759.221274:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495862656:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.222333:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6100, 6100] 00000080:00200000:0.0:1789760759.222358:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.222365:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.222383:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24989696 i_blocks 49159 00000020:00200000:0.0:1789760759.222388:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.222400:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.222515:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24989696, count: 4096 00000080:00200000:0.0:1789760759.222525:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24989696, bytes: 4096 00000080:00200000:0.0:1789760759.222543:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.222553:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.222555:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24989696, 24993792) 0 0 00000080:00200000:0.0:1789760759.222560:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.222577:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6101, 6101] 00020000:00200000:0.0:1789760759.222586:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24989696 chunk: [24989696, 24993792) 24993792, 4096 00020000:00200000:0.0:1789760759.222591:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.222613:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6101, 6101] 00000008:00010000:0.0:1789760759.222633:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.222639:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.222653:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24989696, 24993792) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.222669:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24989696=0x17d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.222741:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.222745:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24993792 00000020:00010000:0.0:1789760759.222889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.222894:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.222898:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.222901:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.222904:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.222912:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.222920:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.222950:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760759.224412:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 24989696<>24993792 00000100:00100000:1.0:1789760759.224535:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495862784:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760759.243815:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760759.276385:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973418 gen 2 00000008:00010000:1.0:1789760759.276432:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760759.276441:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.276452:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.276516:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760759.276545:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495862784:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.277307:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6101, 6101] 00000080:00200000:0.0:1789760759.277325:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.277331:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.277348:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24993792 i_blocks 49159 00000020:00200000:0.0:1789760759.277352:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.277361:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.277397:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24993792, count: 4096 00000080:00200000:0.0:1789760759.277404:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24993792, bytes: 4096 00000080:00200000:0.0:1789760759.277415:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.277422:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.277424:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24993792, 24997888) 0 0 00000080:00200000:0.0:1789760759.277427:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.277441:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6102, 6102] 00020000:00200000:0.0:1789760759.277448:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24993792 chunk: [24993792, 24997888) 24997888, 4096 00020000:00200000:0.0:1789760759.277452:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.277469:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6102, 6102] 00000008:00010000:0.0:1789760759.277497:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.277503:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.277514:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24993792, 24997888) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.277529:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24993792=0x17d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.277671:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.277675:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24997888 00000020:00010000:0.0:1789760759.277684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.277687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.277755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.277760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.277763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.277773:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.277781:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.277901:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760759.278098:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 24993792<>24997888 00000100:00100000:1.0:1789760759.278346:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760759.278407:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495862912:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760759.319458:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760759.319484:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.319497:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.319584:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760759.319617:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495862912:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760759.323796:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6102, 6102] 00000080:00200000:2.0:1789760759.323826:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.323833:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.323853:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24997888 i_blocks 49159 00000020:00200000:2.0:1789760759.323859:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.323875:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.324063:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24997888, count: 4096 00000080:00200000:2.0:1789760759.324075:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24997888, bytes: 4096 00000080:00200000:2.0:1789760759.324089:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.324099:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.324102:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24997888, 25001984) 0 0 00000080:00200000:2.0:1789760759.324107:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.324125:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6103, 6103] 00020000:00200000:2.0:1789760759.324134:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24997888 chunk: [24997888, 25001984) 25001984, 4096 00020000:00200000:2.0:1789760759.324139:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.324164:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6103, 6103] 00000008:00010000:2.0:1789760759.324506:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.324514:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760759.324528:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24997888, 25001984) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.324544:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=24997888=0x17d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.324620:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.324623:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25001984 00000020:00010000:2.0:1789760759.324632:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.324635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.324638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.324642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.324645:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.324654:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.324662:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.324694:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760759.324837:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 24997888<>25001984 00000100:00100000:3.0:1789760759.324952:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495863040:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760759.388123:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973420 gen 2 00000008:00010000:3.0:1789760759.388584:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760759.388595:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.388609:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.388634:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760759.388666:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495863040:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.389958:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6103, 6103] 00000080:00200000:0.0:1789760759.389984:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.389991:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.390013:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25001984 i_blocks 49159 00000020:00200000:0.0:1789760759.390018:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.390034:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.390073:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25001984, count: 4096 00000080:00200000:0.0:1789760759.390080:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25001984, bytes: 4096 00000080:00200000:0.0:1789760759.390097:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.390107:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.390110:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25001984, 25006080) 0 0 00000080:00200000:0.0:1789760759.390115:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.390132:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6104, 6104] 00020000:00200000:0.0:1789760759.390140:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25001984 chunk: [25001984, 25006080) 25006080, 4096 00020000:00200000:0.0:1789760759.390145:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.390206:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6104, 6104] 00000008:00010000:0.0:1789760759.390226:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.390231:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.390243:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25001984, 25006080) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.390258:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25001984=0x17d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.390336:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.390339:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25006080 00000020:00010000:0.0:1789760759.390348:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.390351:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.390355:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.390358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.390361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.390369:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.390377:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.390399:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760759.390543:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 25001984<>25006080 00000100:00100000:0.0:1789760759.390602:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760759.390627:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495863168:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760759.436811:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760759.436827:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.436838:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.436900:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760759.436941:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495863168:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.437214:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6104, 6104] 00000080:00200000:0.0:1789760759.437231:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.437238:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.437255:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25006080 i_blocks 49159 00000020:00200000:0.0:1789760759.437258:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.437267:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.437303:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25006080, count: 4096 00000080:00200000:0.0:1789760759.437311:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25006080, bytes: 4096 00000080:00200000:0.0:1789760759.437321:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.437331:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.437333:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25006080, 25010176) 0 0 00000080:00200000:0.0:1789760759.437338:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.437357:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6105, 6105] 00020000:00200000:0.0:1789760759.437366:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25006080 chunk: [25006080, 25010176) 25010176, 4096 00020000:00200000:0.0:1789760759.437371:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.437393:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6105, 6105] 00000008:00010000:0.0:1789760759.437411:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.437417:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.437429:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25006080, 25010176) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.437444:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25006080=0x17d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.437635:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.437687:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25010176 00000020:00010000:0.0:1789760759.437761:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.437765:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.437768:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.437771:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.437775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.437783:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.437790:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.437810:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760759.438107:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 25006080<>25010176 00000100:00100000:1.0:1789760759.438214:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495863296:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760759.483409:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973421 gen 2 00000100:00100000:1.0:1789760759.483451:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211837800 x1876699495863296/t4294973422(4294973422) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760775 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760759.483512:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760759.483520:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.483530:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.483555:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760759.483579:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495863296:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760759.483681:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6105, 6105] 00000080:00200000:2.0:1789760759.483698:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.483705:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.483721:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25010176 i_blocks 49159 00000020:00200000:2.0:1789760759.483725:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.483734:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.483771:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25010176, count: 4096 00000080:00200000:2.0:1789760759.483778:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25010176, bytes: 4096 00000080:00200000:2.0:1789760759.483788:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.483863:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.483867:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25010176, 25014272) 0 0 00000080:00200000:2.0:1789760759.483871:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.483888:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6106, 6106] 00020000:00200000:2.0:1789760759.483896:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25010176 chunk: [25010176, 25014272) 25014272, 4096 00020000:00200000:2.0:1789760759.483900:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.483921:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6106, 6106] 00000008:00010000:2.0:1789760759.483939:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.483945:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760759.483957:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25010176, 25014272) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.483971:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25010176=0x17da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.484025:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.484028:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25014272 00000020:00010000:2.0:1789760759.484037:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.484041:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.484044:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.484048:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.484051:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.484059:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.484065:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.484080:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760759.489033:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 25010176<>25014272 00000100:00100000:3.0:1789760759.489195:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760759.490940:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495863424:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760759.536698:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973423 gen 2 00000008:00010000:3.0:1789760759.536774:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760759.536783:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.536794:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.536823:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760759.536854:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495863424:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760759.536970:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6106, 6106] 00000080:00200000:1.0:1789760759.536986:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.536993:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.537008:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25014272 i_blocks 49159 00000020:00200000:1.0:1789760759.537012:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.537022:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.537059:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25014272, count: 4096 00000080:00200000:1.0:1789760759.537067:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25014272, bytes: 4096 00000080:00200000:1.0:1789760759.537079:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.537089:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.537091:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25014272, 25018368) 0 0 00000080:00200000:1.0:1789760759.537097:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.537124:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6107, 6107] 00020000:00200000:1.0:1789760759.537133:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25014272 chunk: [25014272, 25018368) 25018368, 4096 00020000:00200000:1.0:1789760759.537138:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.537157:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6107, 6107] 00000008:00010000:1.0:1789760759.537205:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.537211:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760759.537224:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25014272, 25018368) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.537237:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25014272=0x17db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.537293:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.537307:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25018368 00000020:00010000:1.0:1789760759.537315:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.537319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.537322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.537325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.537328:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.537335:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.537342:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.537359:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760759.537511:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 25014272<>25018368 00000100:00100000:1.0:1789760759.537557:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760759.537578:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495863552:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760759.576843:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973424 gen 2 00000008:00010000:1.0:1789760759.576896:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760759.576905:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.576915:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.576935:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760759.576965:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495863552:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.577036:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6107, 6107] 00000080:00200000:0.0:1789760759.577054:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.577061:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.577077:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25018368 i_blocks 49159 00000020:00200000:0.0:1789760759.577081:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.577301:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.577338:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25018368, count: 4096 00000080:00200000:0.0:1789760759.577346:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25018368, bytes: 4096 00000080:00200000:0.0:1789760759.577357:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.577368:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.577371:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25018368, 25022464) 0 0 00000080:00200000:0.0:1789760759.577672:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.577691:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6108, 6108] 00020000:00200000:0.0:1789760759.577700:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25018368 chunk: [25018368, 25022464) 25022464, 4096 00020000:00200000:0.0:1789760759.577705:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.577728:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6108, 6108] 00000008:00010000:0.0:1789760759.577745:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.577751:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.577767:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25018368, 25022464) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.577782:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25018368=0x17dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.577844:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.577847:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25022464 00000020:00010000:0.0:1789760759.577856:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.577859:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.577862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.577866:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.577869:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.577877:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.577886:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.577909:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760759.578056:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 25018368<>25022464 00000100:00100000:0.0:1789760759.578106:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760759.578127:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495863680:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760759.618064:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973425 gen 2 00000008:00010000:1.0:1789760759.618113:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760759.618122:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.618134:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.618159:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760759.618229:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495863680:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.618325:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6108, 6108] 00000080:00200000:0.0:1789760759.618344:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.618352:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.618368:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25022464 i_blocks 49159 00000020:00200000:0.0:1789760759.618371:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.618380:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.618414:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25022464, count: 4096 00000080:00200000:0.0:1789760759.618420:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25022464, bytes: 4096 00000080:00200000:0.0:1789760759.618430:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.618437:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.618439:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25022464, 25026560) 0 0 00000080:00200000:0.0:1789760759.618443:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.618458:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6109, 6109] 00020000:00200000:0.0:1789760759.618466:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25022464 chunk: [25022464, 25026560) 25026560, 4096 00020000:00200000:0.0:1789760759.618470:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.618507:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6109, 6109] 00000008:00010000:0.0:1789760759.618528:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.618534:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.618547:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25022464, 25026560) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.618562:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25022464=0x17dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.618682:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.618685:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25026560 00000020:00010000:0.0:1789760759.618695:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.618699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.618702:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.618706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.618710:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.618717:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.618724:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.618740:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760759.619071:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 25022464<>25026560 00000100:00100000:0.0:1789760759.619317:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495863808:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760759.663084:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760759.663101:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.663114:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.663270:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760759.663300:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495863808:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.663576:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6109, 6109] 00000080:00200000:0.0:1789760759.663593:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.663599:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.663616:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25026560 i_blocks 49159 00000020:00200000:0.0:1789760759.663619:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.663632:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.663668:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25026560, count: 4096 00000080:00200000:0.0:1789760759.663676:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25026560, bytes: 4096 00000080:00200000:0.0:1789760759.663687:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.663696:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.663698:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25026560, 25030656) 0 0 00000080:00200000:0.0:1789760759.663703:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.663720:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6110, 6110] 00020000:00200000:0.0:1789760759.663728:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25026560 chunk: [25026560, 25030656) 25030656, 4096 00020000:00200000:0.0:1789760759.663733:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.664274:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6110, 6110] 00000008:00010000:0.0:1789760759.664295:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.664301:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.664312:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25026560, 25030656) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.664328:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25026560=0x17de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.664391:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.664394:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25030656 00000020:00010000:0.0:1789760759.664402:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.664405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.664410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.664413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.664415:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.664424:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.664431:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.664453:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760759.664600:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 25026560<>25030656 00000100:00100000:1.0:1789760759.664683:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495863936:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760759.715730:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973426 gen 2 00000100:00100000:1.0:1789760759.715783:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211836680 x1876699495863936/t4294973427(4294973427) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760775 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760759.716042:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760759.716049:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.716061:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.716087:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760759.716115:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495863936:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760759.717095:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6110, 6110] 00000080:00200000:2.0:1789760759.717117:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.717124:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.717141:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25030656 i_blocks 49159 00000020:00200000:2.0:1789760759.717145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.717157:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.717301:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25030656, count: 4096 00000080:00200000:2.0:1789760759.717310:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25030656, bytes: 4096 00000080:00200000:2.0:1789760759.717323:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.717333:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.717335:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25030656, 25034752) 0 0 00000080:00200000:2.0:1789760759.717340:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.717357:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6111, 6111] 00020000:00200000:2.0:1789760759.717365:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25030656 chunk: [25030656, 25034752) 25034752, 4096 00020000:00200000:2.0:1789760759.717370:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.717515:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6111, 6111] 00000008:00010000:2.0:1789760759.717536:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.717541:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760759.717553:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25030656, 25034752) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.717569:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25030656=0x17df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.718152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.718158:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25034752 00000020:00010000:2.0:1789760759.718301:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.718306:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.718310:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.718314:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.718316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.718326:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.718335:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.718355:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760759.718629:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 25030656<>25034752 00000100:00100000:3.0:1789760759.718713:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495864064:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760759.768107:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973427 gen 2 00000100:00100000:3.0:1789760759.768163:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264000 x1876699495864064/t4294973428(4294973428) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760775 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760759.768515:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760759.768522:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.768532:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.768596:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760759.768618:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495864064:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760759.769329:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6111, 6111] 00000080:00200000:2.0:1789760759.769352:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.769358:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.769372:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25034752 i_blocks 49159 00000020:00200000:2.0:1789760759.769376:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.769385:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.769418:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25034752, count: 4096 00000080:00200000:2.0:1789760759.769425:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25034752, bytes: 4096 00000080:00200000:2.0:1789760759.769436:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.769444:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.769447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25034752, 25038848) 0 0 00000080:00200000:2.0:1789760759.769520:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.769537:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6112, 6112] 00020000:00200000:2.0:1789760759.769546:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25034752 chunk: [25034752, 25038848) 25038848, 4096 00020000:00200000:2.0:1789760759.769552:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.769574:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6112, 6112] 00000008:00010000:2.0:1789760759.769593:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.769605:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760759.769618:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25034752, 25038848) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.769633:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25034752=0x17e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.769696:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.769699:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25038848 00000020:00010000:2.0:1789760759.769707:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.769711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.769714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.769718:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.769720:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.769728:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.769735:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.770031:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760759.770164:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 25034752<>25038848 00000100:00100000:2.0:1789760759.770434:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495864192:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760759.808344:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973428 gen 2 00000100:00100000:2.0:1789760759.808382:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266d80 x1876699495864192/t4294973429(4294973429) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760775 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760759.808426:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760759.808433:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.808443:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.808521:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760759.808544:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495864192:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760759.808658:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6112, 6112] 00000080:00200000:3.0:1789760759.808674:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.808682:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.808697:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25038848 i_blocks 49159 00000020:00200000:3.0:1789760759.808700:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.808710:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.808742:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25038848, count: 4096 00000080:00200000:3.0:1789760759.808749:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25038848, bytes: 4096 00000080:00200000:3.0:1789760759.808760:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.808769:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.808771:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25038848, 25042944) 0 0 00000080:00200000:3.0:1789760759.808775:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.808801:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6113, 6113] 00020000:00200000:3.0:1789760759.808809:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25038848 chunk: [25038848, 25042944) 25042944, 4096 00020000:00200000:3.0:1789760759.808813:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.808832:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6113, 6113] 00000008:00010000:3.0:1789760759.808849:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.808855:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760759.808867:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25038848, 25042944) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.808881:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25038848=0x17e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.808936:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.808938:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25042944 00000020:00010000:3.0:1789760759.808946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.808949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.808952:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.808956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.808959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.808965:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.808972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.808987:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760759.809108:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 25038848<>25042944 00000100:00100000:2.0:1789760759.809380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495864320:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760759.856680:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973429 gen 2 00000100:00100000:2.0:1789760759.856722:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203145c00 x1876699495864320/t4294973430(4294973430) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760775 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760759.856771:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760759.856779:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.856790:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.856933:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760759.856959:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495864320:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760759.857202:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6113, 6113] 00000080:00200000:1.0:1789760759.857221:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.857227:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.857242:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25042944 i_blocks 49159 00000020:00200000:1.0:1789760759.857246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.857256:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.857294:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25042944, count: 4096 00000080:00200000:1.0:1789760759.857301:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25042944, bytes: 4096 00000080:00200000:1.0:1789760759.857314:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.857323:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.857326:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25042944, 25047040) 0 0 00000080:00200000:1.0:1789760759.857330:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.857348:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6114, 6114] 00020000:00200000:1.0:1789760759.857356:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25042944 chunk: [25042944, 25047040) 25047040, 4096 00020000:00200000:1.0:1789760759.857361:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.857382:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6114, 6114] 00000008:00010000:1.0:1789760759.857673:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.857680:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760759.857693:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25042944, 25047040) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.857708:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25042944=0x17e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.857769:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.857772:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25047040 00000020:00010000:1.0:1789760759.857780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.857784:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.857786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.857790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.857863:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.857872:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.857879:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.857901:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760759.858116:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 25042944<>25047040 00000100:00100000:0.0:1789760759.858200:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495864448:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760759.909902:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973430 gen 2 00000100:00100000:1.0:1789760759.909948:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504df80 x1876699495864448/t4294973431(4294973431) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760775 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760759.909998:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760759.910006:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.910016:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.910090:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000080:00200000:3.0:1789760759.910097:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6114, 6114] 00000100:00100000:1.0:1789760759.910117:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495864448:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760759.910118:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.910124:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.910148:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25047040 i_blocks 49159 00000020:00200000:3.0:1789760759.910152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.910161:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.910226:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25047040, count: 4096 00000080:00200000:3.0:1789760759.910266:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25047040, bytes: 4096 00000080:00200000:3.0:1789760759.910281:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.910291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.910293:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25047040, 25051136) 0 0 00000080:00200000:3.0:1789760759.910297:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.910316:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6115, 6115] 00020000:00200000:3.0:1789760759.910324:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25047040 chunk: [25047040, 25051136) 25051136, 4096 00020000:00200000:3.0:1789760759.910329:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.910402:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6115, 6115] 00000008:00010000:3.0:1789760759.910418:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.910423:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760759.910434:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25047040, 25051136) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.910446:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25047040=0x17e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.910567:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.910570:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25051136 00000020:00010000:3.0:1789760759.910580:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.910583:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.910586:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.910590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.910593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.910601:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.910612:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.910629:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760759.910757:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 25047040<>25051136 00000100:00100000:2.0:1789760759.910845:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495864576:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760759.969383:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973432 gen 2 00000008:00010000:2.0:1789760759.969446:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:2.0:1789760759.969455:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.969486:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.969577:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:2.0:1789760759.969604:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495864576:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760759.969774:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6115, 6115] 00000080:00200000:0.0:1789760759.969799:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.969806:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.969821:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25051136 i_blocks 49159 00000020:00200000:0.0:1789760759.969825:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.969833:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.969936:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25051136, count: 4096 00000080:00200000:0.0:1789760759.969944:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25051136, bytes: 4096 00000080:00200000:0.0:1789760759.969957:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.969966:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.969968:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25051136, 25055232) 0 0 00000080:00200000:0.0:1789760759.969973:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.970281:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6116, 6116] 00020000:00200000:0.0:1789760759.970291:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25051136 chunk: [25051136, 25055232) 25055232, 4096 00020000:00200000:0.0:1789760759.970296:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.970320:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6116, 6116] 00000008:00010000:0.0:1789760759.970341:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.970347:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760759.970359:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25051136, 25055232) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.970374:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25051136=0x17e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.970431:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.970434:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25055232 00000020:00010000:0.0:1789760759.970443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.970446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.970450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.970453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.970516:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.970527:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.970534:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.970556:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760759.970688:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 25051136<>25055232 00000100:00100000:1.0:1789760759.970785:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495864704:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760760.040730:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973433 gen 2 00000008:00010000:1.0:1789760760.040781:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760760.040789:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.040801:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.040832:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760760.040864:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495864704:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.040995:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6116, 6116] 00000080:00200000:2.0:1789760760.041012:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.041018:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.041032:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25055232 i_blocks 49159 00000020:00200000:2.0:1789760760.041036:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.041046:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.041080:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25055232, count: 4096 00000080:00200000:2.0:1789760760.041087:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25055232, bytes: 4096 00000080:00200000:2.0:1789760760.041098:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.041107:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.041109:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25055232, 25059328) 0 0 00000080:00200000:2.0:1789760760.041113:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.041131:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6117, 6117] 00020000:00200000:2.0:1789760760.041139:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25055232 chunk: [25055232, 25059328) 25059328, 4096 00020000:00200000:2.0:1789760760.041144:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.041165:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6117, 6117] 00000008:00010000:2.0:1789760760.041215:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.041221:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.041233:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25055232, 25059328) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.041249:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25055232=0x17e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.041303:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.041306:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25059328 00000020:00010000:2.0:1789760760.041313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.041317:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.041320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.041324:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.041327:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.041334:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.041341:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.041356:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760760.041484:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 25055232<>25059328 00000100:00100000:2.0:1789760760.041527:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760760.041543:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495864832:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760760.079094:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973434 gen 2 00000008:00010000:2.0:1789760760.079146:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760760.079155:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.079209:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.079245:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760760.079274:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495864832:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760760.079512:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6117, 6117] 00000080:00200000:1.0:1789760760.079531:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.079537:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.079553:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25059328 i_blocks 49159 00000020:00200000:1.0:1789760760.079556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.079566:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.079602:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25059328, count: 4096 00000080:00200000:1.0:1789760760.079609:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25059328, bytes: 4096 00000080:00200000:1.0:1789760760.079625:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.079634:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.079636:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25059328, 25063424) 0 0 00000080:00200000:1.0:1789760760.079641:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.079658:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6118, 6118] 00020000:00200000:1.0:1789760760.079666:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25059328 chunk: [25059328, 25063424) 25063424, 4096 00020000:00200000:1.0:1789760760.079671:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.079692:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6118, 6118] 00000008:00010000:1.0:1789760760.079709:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.079715:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760760.079726:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25059328, 25063424) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.079741:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25059328=0x17e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.079799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.079802:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25063424 00000020:00010000:1.0:1789760760.079811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.079814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.079817:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.079821:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.079824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.079833:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.079840:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.079857:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760760.079983:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 25059328<>25063424 00000100:00100000:1.0:1789760760.080042:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760760.080059:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495864960:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760760.125351:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973435 gen 2 00000008:00010000:1.0:1789760760.125406:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760760.125415:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.125426:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.125451:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760760.125490:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495864960:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.125618:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6118, 6118] 00000080:00200000:2.0:1789760760.125635:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.125640:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.125658:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25063424 i_blocks 49159 00000020:00200000:2.0:1789760760.125662:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.125672:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.125709:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25063424, count: 4096 00000080:00200000:2.0:1789760760.125715:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25063424, bytes: 4096 00000080:00200000:2.0:1789760760.125725:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.125736:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.125739:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25063424, 25067520) 0 0 00000080:00200000:2.0:1789760760.125744:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.125762:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6119, 6119] 00020000:00200000:2.0:1789760760.125769:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25063424 chunk: [25063424, 25067520) 25067520, 4096 00020000:00200000:2.0:1789760760.125774:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.125805:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6119, 6119] 00000008:00010000:2.0:1789760760.125823:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.125828:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.125839:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25063424, 25067520) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.125853:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25063424=0x17e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.125909:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.125912:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25067520 00000020:00010000:2.0:1789760760.125921:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.125925:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.125928:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.125932:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.125935:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.125942:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.125949:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.125966:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760760.126093:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 25063424<>25067520 00000100:00100000:3.0:1789760760.126488:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495865088:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.192579:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973436 gen 2 00000008:00010000:3.0:1789760760.192627:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760760.192635:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.192647:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.192667:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760760.192696:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495865088:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.192716:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6119, 6119] 00000080:00200000:2.0:1789760760.192735:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.192742:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.192758:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25067520 i_blocks 49159 00000020:00200000:2.0:1789760760.192762:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.192772:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.192807:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25067520, count: 4096 00000080:00200000:2.0:1789760760.192814:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25067520, bytes: 4096 00000080:00200000:2.0:1789760760.192824:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.192833:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.192836:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25067520, 25071616) 0 0 00000080:00200000:2.0:1789760760.192840:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.192858:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6120, 6120] 00020000:00200000:2.0:1789760760.192866:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25067520 chunk: [25067520, 25071616) 25071616, 4096 00020000:00200000:2.0:1789760760.192871:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.192890:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6120, 6120] 00000008:00010000:2.0:1789760760.192910:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.192916:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.192926:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25067520, 25071616) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.192942:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25067520=0x17e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.193005:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.193008:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25071616 00000020:00010000:2.0:1789760760.193017:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.193020:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.193023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.193027:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.193030:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.193036:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.193043:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.193060:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760760.193226:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 25067520<>25071616 00000100:00100000:2.0:1789760760.193281:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760760.193296:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495865216:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760760.254509:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760760.254527:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.254540:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.254562:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760760.254594:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495865216:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760760.255043:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6120, 6120] 00000080:00200000:3.0:1789760760.255063:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.255071:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.255089:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25071616 i_blocks 49159 00000020:00200000:3.0:1789760760.255093:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.255104:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.255138:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25071616, count: 4096 00000080:00200000:3.0:1789760760.255145:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25071616, bytes: 4096 00000080:00200000:3.0:1789760760.255157:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.255165:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.255204:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25071616, 25075712) 0 0 00000080:00200000:3.0:1789760760.255210:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.255228:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6121, 6121] 00020000:00200000:3.0:1789760760.255237:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25071616 chunk: [25071616, 25075712) 25075712, 4096 00020000:00200000:3.0:1789760760.255242:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.255263:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6121, 6121] 00000008:00010000:3.0:1789760760.255285:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.255292:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760760.255305:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25071616, 25075712) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.255321:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25071616=0x17e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.255384:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.255387:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25075712 00000020:00010000:3.0:1789760760.255397:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.255400:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.255405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.255408:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.255411:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.255418:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.255425:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.255456:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760760.255594:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 25071616<>25075712 00000100:00100000:3.0:1789760760.255640:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760760.255658:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495865344:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760760.267364:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760760.290255:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973437 gen 2 00000100:00100000:3.0:1789760760.290298:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203144a80 x1876699495865344/t4294973438(4294973438) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760776 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760760.290347:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:3.0:1789760760.290354:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.290365:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.290389:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:3.0:1789760760.290414:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495865344:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.290517:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6121, 6121] 00000080:00200000:2.0:1789760760.290536:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.290541:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.290556:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25075712 i_blocks 49159 00000020:00200000:2.0:1789760760.290559:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.290568:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.290601:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25075712, count: 4096 00000080:00200000:2.0:1789760760.290608:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25075712, bytes: 4096 00000080:00200000:2.0:1789760760.290617:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.290625:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.290627:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25075712, 25079808) 0 0 00000080:00200000:2.0:1789760760.290631:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.290702:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6122, 6122] 00020000:00200000:2.0:1789760760.290711:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25075712 chunk: [25075712, 25079808) 25079808, 4096 00020000:00200000:2.0:1789760760.290715:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.290735:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6122, 6122] 00000008:00010000:2.0:1789760760.290753:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.290759:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.290770:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25075712, 25079808) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.290785:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25075712=0x17ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.290841:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.290844:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25079808 00000020:00010000:2.0:1789760760.290852:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.290856:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.290858:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.290862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.290865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.290872:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.290879:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.290895:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760760.292137:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 25075712<>25079808 00000100:00100000:3.0:1789760760.292255:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495865472:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.337407:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973439 gen 2 00000008:00010000:3.0:1789760760.337493:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760760.337502:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.337513:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.337540:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000080:00200000:0.0:1789760760.337565:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6122, 6122] 00000100:00100000:3.0:1789760760.337573:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495865472:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760760.337598:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.337605:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.337624:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25079808 i_blocks 49159 00000020:00200000:0.0:1789760760.337628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.337639:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.337678:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25079808, count: 4096 00000080:00200000:0.0:1789760760.337685:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25079808, bytes: 4096 00000080:00200000:0.0:1789760760.337696:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.337705:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.337707:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25079808, 25083904) 0 0 00000080:00200000:0.0:1789760760.337712:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.337730:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6123, 6123] 00020000:00200000:0.0:1789760760.337738:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25079808 chunk: [25079808, 25083904) 25083904, 4096 00020000:00200000:0.0:1789760760.337743:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.337763:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6123, 6123] 00000008:00010000:0.0:1789760760.337783:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.337789:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760760.338300:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25079808, 25083904) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.338314:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25079808=0x17eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.338362:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.338365:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25083904 00000020:00010000:0.0:1789760760.338371:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.338374:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.338377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.338380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.338384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.338392:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.338399:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.338416:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760760.338674:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 25079808<>25083904 00000100:00100000:1.0:1789760760.338753:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495865600:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760760.376313:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760760.376329:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.376340:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.376496:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760760.376526:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495865600:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760760.377905:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6123, 6123] 00000080:00200000:0.0:1789760760.377929:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.377936:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.377953:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25083904 i_blocks 49159 00000020:00200000:0.0:1789760760.377958:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.378099:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.378209:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25083904, count: 4096 00000080:00200000:0.0:1789760760.378218:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25083904, bytes: 4096 00000080:00200000:0.0:1789760760.378233:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.378242:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.378244:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25083904, 25088000) 0 0 00000080:00200000:0.0:1789760760.378249:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.378267:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6124, 6124] 00020000:00200000:0.0:1789760760.378275:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25083904 chunk: [25083904, 25088000) 25088000, 4096 00020000:00200000:0.0:1789760760.378279:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.378301:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6124, 6124] 00000008:00010000:0.0:1789760760.378318:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.378325:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760760.378336:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25083904, 25088000) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.378349:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25083904=0x17ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.378519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.378523:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25088000 00000020:00010000:0.0:1789760760.378533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.378537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.378540:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.378543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.378546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.378555:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.378561:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.378589:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760760.378922:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 25083904<>25088000 00000100:00100000:1.0:1789760760.379119:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495865728:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760760.412541:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973441 gen 2 00000008:00010000:1.0:1789760760.412608:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760760.412616:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.412626:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.412698:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760760.412733:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495865728:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.412941:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6124, 6124] 00000080:00200000:2.0:1789760760.412958:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.412964:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.412981:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25088000 i_blocks 49159 00000020:00200000:2.0:1789760760.412984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.412994:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.413097:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25088000, count: 4096 00000080:00200000:2.0:1789760760.413104:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25088000, bytes: 4096 00000080:00200000:2.0:1789760760.414851:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.414863:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.414866:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25088000, 25092096) 0 0 00000080:00200000:2.0:1789760760.414872:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.414893:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6125, 6125] 00020000:00200000:2.0:1789760760.414903:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25088000 chunk: [25088000, 25092096) 25092096, 4096 00020000:00200000:2.0:1789760760.414908:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.414933:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6125, 6125] 00000008:00010000:2.0:1789760760.414953:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.414959:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.414971:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25088000, 25092096) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.414987:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25088000=0x17ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.415065:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.415069:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25092096 00000020:00010000:2.0:1789760760.415078:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.415082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.415085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.415089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.415092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.415103:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.415120:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.415141:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760760.415309:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 25088000<>25092096 00000100:00100000:3.0:1789760760.415515:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495865856:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.460417:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973442 gen 2 00000008:00010000:3.0:1789760760.460486:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760760.460495:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.460508:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.460534:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760760.460562:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495865856:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.460780:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6125, 6125] 00000080:00200000:2.0:1789760760.460809:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.460816:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.460832:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25092096 i_blocks 49159 00000020:00200000:2.0:1789760760.460836:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.460856:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.460892:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25092096, count: 4096 00000080:00200000:2.0:1789760760.460899:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25092096, bytes: 4096 00000080:00200000:2.0:1789760760.460909:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.460918:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.460920:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25092096, 25096192) 0 0 00000080:00200000:2.0:1789760760.460925:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.460942:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6126, 6126] 00020000:00200000:2.0:1789760760.460950:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25092096 chunk: [25092096, 25096192) 25096192, 4096 00020000:00200000:2.0:1789760760.460954:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.460976:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6126, 6126] 00000008:00010000:2.0:1789760760.460995:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.461001:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.461012:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25092096, 25096192) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.461026:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25092096=0x17ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.461081:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.461084:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25096192 00000020:00010000:2.0:1789760760.461092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.461096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.461099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.461102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.461105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.461112:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.461119:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.461136:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760760.461293:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 25092096<>25096192 00000100:00100000:3.0:1789760760.461575:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760760.461594:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495865984:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.494079:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973443 gen 2 00000008:00010000:3.0:1789760760.494133:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760760.494141:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.494151:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.494205:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760760.494235:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495865984:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.494670:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6126, 6126] 00000080:00200000:2.0:1789760760.494688:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.494695:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.494711:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25096192 i_blocks 49159 00000020:00200000:2.0:1789760760.494715:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.494726:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.494761:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25096192, count: 4096 00000080:00200000:2.0:1789760760.494768:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25096192, bytes: 4096 00000080:00200000:2.0:1789760760.494779:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.494796:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.494799:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25096192, 25100288) 0 0 00000080:00200000:2.0:1789760760.494804:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.494821:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6127, 6127] 00020000:00200000:2.0:1789760760.494829:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25096192 chunk: [25096192, 25100288) 25100288, 4096 00020000:00200000:2.0:1789760760.494835:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.494864:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6127, 6127] 00000008:00010000:2.0:1789760760.494883:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.494888:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.494900:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25096192, 25100288) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.494914:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25096192=0x17ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.494970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.494973:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25100288 00000020:00010000:2.0:1789760760.494981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.494985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.494988:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.494992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.494995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.495003:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.495010:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.495026:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760760.495143:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 25096192<>25100288 00000100:00100000:3.0:1789760760.495416:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495866112:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.524003:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973444 gen 2 00000008:00010000:3.0:1789760760.524055:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760760.524064:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.524076:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.524104:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760760.524132:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495866112:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.524368:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6127, 6127] 00000080:00200000:2.0:1789760760.524386:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.524392:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.524407:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25100288 i_blocks 49159 00000020:00200000:2.0:1789760760.524410:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.524421:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.524456:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25100288, count: 4096 00000080:00200000:2.0:1789760760.524463:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25100288, bytes: 4096 00000080:00200000:2.0:1789760760.524480:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.524488:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.524490:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25100288, 25104384) 0 0 00000080:00200000:2.0:1789760760.524495:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.524516:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6128, 6128] 00020000:00200000:2.0:1789760760.524524:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25100288 chunk: [25100288, 25104384) 25104384, 4096 00020000:00200000:2.0:1789760760.524529:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.524552:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6128, 6128] 00000008:00010000:2.0:1789760760.524570:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.524575:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.524588:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25100288, 25104384) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.524601:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25100288=0x17f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.524720:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.524723:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25104384 00000020:00010000:2.0:1789760760.524733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.524736:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.524740:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.524743:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.524747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.524754:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.524761:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.524780:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760760.525404:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 25100288<>25104384 00000100:00100000:3.0:1789760760.525490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495866240:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.566716:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973445 gen 2 00000008:00010000:3.0:1789760760.566773:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760760.566781:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.566861:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.566888:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760760.566919:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495866240:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760760.567057:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6128, 6128] 00000080:00200000:0.0:1789760760.567074:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.567081:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.567097:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25104384 i_blocks 49159 00000020:00200000:0.0:1789760760.567101:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.567110:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.567143:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25104384, count: 4096 00000080:00200000:0.0:1789760760.567151:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25104384, bytes: 4096 00000080:00200000:0.0:1789760760.567162:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.567201:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.567204:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25104384, 25108480) 0 0 00000080:00200000:0.0:1789760760.567209:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.567227:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6129, 6129] 00020000:00200000:0.0:1789760760.567236:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25104384 chunk: [25104384, 25108480) 25108480, 4096 00020000:00200000:0.0:1789760760.567240:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.567262:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6129, 6129] 00000008:00010000:0.0:1789760760.567280:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.567286:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760760.567298:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25104384, 25108480) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.567312:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25104384=0x17f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.567702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.567706:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25108480 00000020:00010000:0.0:1789760760.567715:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.567719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.567722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.567727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.567730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.567739:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.567746:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.567766:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760760.569067:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 25104384<>25108480 00000100:00100000:0.0:1789760760.569144:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495866368:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760760.607209:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760760.607225:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.607236:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.607262:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760760.607292:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495866368:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760760.607306:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6129, 6129] 00000080:00200000:3.0:1789760760.607326:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.607332:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.607348:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25108480 i_blocks 49159 00000020:00200000:3.0:1789760760.607352:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.607361:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.607396:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25108480, count: 4096 00000080:00200000:3.0:1789760760.607403:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25108480, bytes: 4096 00000080:00200000:3.0:1789760760.607414:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.607422:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.607424:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25108480, 25112576) 0 0 00000080:00200000:3.0:1789760760.607429:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.607446:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6130, 6130] 00020000:00200000:3.0:1789760760.607459:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25108480 chunk: [25108480, 25112576) 25112576, 4096 00020000:00200000:3.0:1789760760.607471:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.607569:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6130, 6130] 00000008:00010000:1.0:1789760760.607591:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.607597:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760760.607610:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25108480, 25112576) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.607633:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25108480=0x17f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.607690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.607693:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25112576 00000020:00010000:1.0:1789760760.607702:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.607705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.607708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.607711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.607714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.607722:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.607729:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.607742:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760760.607984:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 25108480<>25112576 00000100:00100000:1.0:1789760760.608024:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760760.608038:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495866496:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760760.658829:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973447 gen 2 00000008:00010000:1.0:1789760760.658900:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760760.658909:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.658919:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.658949:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760760.658980:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495866496:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.659081:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6130, 6130] 00000080:00200000:2.0:1789760760.659099:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.659105:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.659120:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25112576 i_blocks 49159 00000020:00200000:2.0:1789760760.659124:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.659133:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.659187:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25112576, count: 4096 00000080:00200000:2.0:1789760760.659194:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25112576, bytes: 4096 00000080:00200000:2.0:1789760760.659203:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.659212:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.659214:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25112576, 25116672) 0 0 00000080:00200000:2.0:1789760760.659218:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.659234:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6131, 6131] 00020000:00200000:2.0:1789760760.659241:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25112576 chunk: [25112576, 25116672) 25116672, 4096 00020000:00200000:2.0:1789760760.659245:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.659263:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6131, 6131] 00000008:00010000:2.0:1789760760.659279:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.659285:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.659295:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25112576, 25116672) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.659309:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25112576=0x17f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.659366:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.659369:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25116672 00000020:00010000:2.0:1789760760.659377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.659381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.659384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.659387:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.659390:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.659397:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.659404:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.659419:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760760.659562:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 25112576<>25116672 00000100:00100000:3.0:1789760760.659703:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495866624:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.705957:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973448 gen 2 00000008:00010000:3.0:1789760760.706009:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760760.706017:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.706098:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.706244:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760760.706275:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495866624:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.706533:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6131, 6131] 00000080:00200000:2.0:1789760760.706554:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.706561:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.706579:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25116672 i_blocks 49159 00000020:00200000:2.0:1789760760.706583:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.706595:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.706634:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25116672, count: 4096 00000080:00200000:2.0:1789760760.706642:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25116672, bytes: 4096 00000080:00200000:2.0:1789760760.706654:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.706664:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.706667:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25116672, 25120768) 0 0 00000080:00200000:2.0:1789760760.706672:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.706693:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6132, 6132] 00020000:00200000:2.0:1789760760.706703:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25116672 chunk: [25116672, 25120768) 25120768, 4096 00020000:00200000:2.0:1789760760.706708:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.706733:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6132, 6132] 00000008:00010000:2.0:1789760760.706752:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.706758:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.706771:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25116672, 25120768) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.706897:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25116672=0x17f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.706957:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.706960:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25120768 00000020:00010000:2.0:1789760760.706968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.706972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.706975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.706979:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.706981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.706988:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.706994:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.707011:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760760.707220:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 25116672<>25120768 00000100:00100000:3.0:1789760760.707322:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760760.707404:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495866752:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760760.743710:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760760.743726:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.743738:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.743827:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760760.743859:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495866752:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.745764:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6132, 6132] 00000080:00200000:2.0:1789760760.746096:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.746105:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.746125:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25120768 i_blocks 49159 00000020:00200000:2.0:1789760760.746130:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.746148:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.746613:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25120768, count: 4096 00000080:00200000:2.0:1789760760.746625:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25120768, bytes: 4096 00000080:00200000:2.0:1789760760.746651:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.746661:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.746663:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25120768, 25124864) 0 0 00000080:00200000:2.0:1789760760.746668:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.746686:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6133, 6133] 00020000:00200000:2.0:1789760760.746695:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25120768 chunk: [25120768, 25124864) 25124864, 4096 00020000:00200000:2.0:1789760760.746700:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.746723:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6133, 6133] 00000008:00010000:2.0:1789760760.746742:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.746746:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.746757:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25120768, 25124864) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.746769:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25120768=0x17f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.746834:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.746836:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25124864 00000020:00010000:2.0:1789760760.746843:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.746845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.746848:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.746851:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.746853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.746861:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.746869:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.746897:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760760.747050:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 25120768<>25124864 00000100:00100000:3.0:1789760760.747455:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495866880:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.790300:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973450 gen 2 00000008:00010000:3.0:1789760760.790362:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760760.790370:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.790381:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.790579:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000080:00200000:2.0:1789760760.790587:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6133, 6133] 00000080:00200000:2.0:1789760760.790606:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760760.790612:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495866880:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.790613:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.790629:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25124864 i_blocks 49159 00000020:00200000:2.0:1789760760.790632:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.790642:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.790677:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25124864, count: 4096 00000080:00200000:2.0:1789760760.790686:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25124864, bytes: 4096 00000080:00200000:2.0:1789760760.790761:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.790772:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.790775:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25124864, 25128960) 0 0 00000080:00200000:2.0:1789760760.790780:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.790944:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6134, 6134] 00020000:00200000:2.0:1789760760.790952:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25124864 chunk: [25124864, 25128960) 25128960, 4096 00020000:00200000:2.0:1789760760.790955:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.790974:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6134, 6134] 00000008:00010000:2.0:1789760760.790988:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.790993:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.791003:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25124864, 25128960) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.791015:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25124864=0x17f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.791067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.791071:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25128960 00000020:00010000:2.0:1789760760.791079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.791083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.791086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.791089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.791092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.791100:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.791106:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.791123:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760760.791396:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 25124864<>25128960 00000100:00100000:3.0:1789760760.791473:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495867008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760760.836069:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973451 gen 2 00000008:00010000:3.0:1789760760.836119:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760760.836127:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.836139:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.836269:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760760.836297:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495867008:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760760.836520:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6134, 6134] 00000080:00200000:1.0:1789760760.836539:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.836611:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.836629:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25128960 i_blocks 49159 00000020:00200000:1.0:1789760760.836633:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.836645:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.836680:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25128960, count: 4096 00000080:00200000:1.0:1789760760.836687:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25128960, bytes: 4096 00000080:00200000:1.0:1789760760.836699:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.836707:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.836709:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25128960, 25133056) 0 0 00000080:00200000:1.0:1789760760.836714:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.836732:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6135, 6135] 00020000:00200000:1.0:1789760760.836741:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25128960 chunk: [25128960, 25133056) 25133056, 4096 00020000:00200000:1.0:1789760760.836747:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.836768:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6135, 6135] 00000008:00010000:1.0:1789760760.836904:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.836911:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760760.836926:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25128960, 25133056) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.836941:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25128960=0x17f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.837003:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.837006:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25133056 00000020:00010000:1.0:1789760760.837015:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.837018:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.837022:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.837025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.837028:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.837035:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.837042:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.837060:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760760.837223:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 25128960<>25133056 00000100:00100000:0.0:1789760760.837311:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495867136:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760760.881264:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973452 gen 2 00000008:00010000:1.0:1789760760.881518:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760760.881528:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.881539:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.881560:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760760.881592:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495867136:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760760.881706:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6135, 6135] 00000080:00200000:3.0:1789760760.881724:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.881730:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.881747:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25133056 i_blocks 49159 00000020:00200000:3.0:1789760760.881750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.881760:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.881814:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25133056, count: 4096 00000080:00200000:3.0:1789760760.881821:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25133056, bytes: 4096 00000080:00200000:3.0:1789760760.881833:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.881842:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.881844:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25133056, 25137152) 0 0 00000080:00200000:3.0:1789760760.881858:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.881876:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6136, 6136] 00020000:00200000:3.0:1789760760.881886:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25133056 chunk: [25133056, 25137152) 25137152, 4096 00020000:00200000:3.0:1789760760.881890:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.881910:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6136, 6136] 00000008:00010000:3.0:1789760760.881928:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.881934:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760760.881946:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25133056, 25137152) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.881959:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25133056=0x17f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.882017:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.882019:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25137152 00000020:00010000:3.0:1789760760.882028:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.882032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.882035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.882039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.882042:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.882049:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.882057:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.882073:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760760.882247:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 25133056<>25137152 00000100:00100000:3.0:1789760760.882297:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760760.882314:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495867264:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760760.923951:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973453 gen 2 00000008:00010000:2.0:1789760760.923998:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760760.924006:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.924017:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.924163:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760760.924216:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495867264:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760760.924318:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6136, 6136] 00000080:00200000:0.0:1789760760.924404:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.924412:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.924429:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25137152 i_blocks 49159 00000020:00200000:0.0:1789760760.924433:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.924556:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.924593:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25137152, count: 4096 00000080:00200000:0.0:1789760760.924599:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25137152, bytes: 4096 00000080:00200000:0.0:1789760760.924609:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.924618:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.924621:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25137152, 25141248) 0 0 00000080:00200000:0.0:1789760760.924626:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.924643:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6137, 6137] 00020000:00200000:0.0:1789760760.924652:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25137152 chunk: [25137152, 25141248) 25141248, 4096 00020000:00200000:0.0:1789760760.924657:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.924675:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6137, 6137] 00000008:00010000:0.0:1789760760.924692:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.924698:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760760.924711:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25137152, 25141248) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.924725:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25137152=0x17f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.924851:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.924854:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25141248 00000020:00010000:0.0:1789760760.924865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.924869:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.924872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.924875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.924878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.924886:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.924893:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.924912:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760760.925413:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 25137152<>25141248 00000100:00100000:1.0:1789760760.925479:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760760.925640:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495867392:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760760.969560:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973454 gen 2 00000008:00010000:1.0:1789760760.969611:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760760.969620:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.969633:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.969662:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760760.969693:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495867392:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760760.969823:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6137, 6137] 00000080:00200000:2.0:1789760760.969842:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.969848:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.969864:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25141248 i_blocks 49159 00000020:00200000:2.0:1789760760.969867:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.969877:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.969911:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25141248, count: 4096 00000080:00200000:2.0:1789760760.969919:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25141248, bytes: 4096 00000080:00200000:2.0:1789760760.969928:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.969938:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.969940:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25141248, 25145344) 0 0 00000080:00200000:2.0:1789760760.969946:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.969962:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6138, 6138] 00020000:00200000:2.0:1789760760.969970:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25141248 chunk: [25141248, 25145344) 25145344, 4096 00020000:00200000:2.0:1789760760.969975:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.969996:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6138, 6138] 00000008:00010000:2.0:1789760760.970202:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.970209:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760760.970287:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25141248, 25145344) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.970304:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25141248=0x17fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.970363:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.970366:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25145344 00000020:00010000:2.0:1789760760.970374:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.970377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.970380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.970383:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.970385:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.970393:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.970401:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.970419:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760760.970559:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 25141248<>25145344 00000100:00100000:3.0:1789760760.970638:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495867520:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760761.015932:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973455 gen 2 00000008:00010000:2.0:1789760761.015983:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760761.015991:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.016003:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.016123:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760761.016155:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495867520:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760761.016844:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6138, 6138] 00000080:00200000:1.0:1789760761.016864:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760761.016872:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760761.016887:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25145344 i_blocks 49159 00000020:00200000:1.0:1789760761.016892:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760761.016903:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760761.016939:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25145344, count: 4096 00000080:00200000:1.0:1789760761.016946:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25145344, bytes: 4096 00000080:00200000:1.0:1789760761.016957:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760761.016965:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760761.016967:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25145344, 25149440) 0 0 00000080:00200000:1.0:1789760761.016971:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760761.016986:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6139, 6139] 00020000:00200000:1.0:1789760761.016994:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25145344 chunk: [25145344, 25149440) 25149440, 4096 00020000:00200000:1.0:1789760761.016998:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760761.017016:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6139, 6139] 00000008:00010000:1.0:1789760761.017033:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760761.017038:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760761.017049:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25145344, 25149440) append 0 append lockpos 0 00000080:00200000:1.0:1789760761.017065:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25145344=0x17fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760761.017123:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760761.017125:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25149440 00000020:00010000:1.0:1789760761.017133:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760761.017136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.017139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.017143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760761.017146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760761.017153:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760761.017159:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760761.017209:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760761.017321:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 25145344<>25149440 00000100:00100000:0.0:1789760761.017565:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495867648:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760761.067317:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760761.067335:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.067346:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.067974:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760761.068008:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495867648:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.069159:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6139, 6139] 00000080:00200000:2.0:1789760761.069202:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.069208:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.069251:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25149440 i_blocks 49159 00000020:00200000:2.0:1789760761.069256:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.069268:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.069303:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25149440, count: 4096 00000080:00200000:2.0:1789760761.069310:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25149440, bytes: 4096 00000080:00200000:2.0:1789760761.069324:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.069333:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.069335:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25149440, 25153536) 0 0 00000080:00200000:2.0:1789760761.069340:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.069357:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6140, 6140] 00020000:00200000:2.0:1789760761.069365:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25149440 chunk: [25149440, 25153536) 25153536, 4096 00020000:00200000:2.0:1789760761.069371:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.069394:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6140, 6140] 00000008:00010000:2.0:1789760761.069421:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.069427:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.069440:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25149440, 25153536) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.069455:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25149440=0x17fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.069523:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.069527:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25153536 00000020:00010000:2.0:1789760761.069536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.069539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.069542:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.069546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.069549:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.069558:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.069565:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.069579:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760761.069713:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 25149440<>25153536 00000100:00100000:2.0:1789760761.069776:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.069798:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495867776:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760761.122677:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973456 gen 2 00000100:00100000:2.0:1789760761.122721:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265500 x1876699495867776/t4294973457(4294973457) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760777 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760761.122774:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760761.122782:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.122794:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.122831:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760761.122860:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495867776:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760761.123782:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6140, 6140] 00000080:00200000:1.0:1789760761.123812:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760761.123820:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760761.123838:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25153536 i_blocks 49159 00000020:00200000:1.0:1789760761.123842:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760761.123854:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760761.123999:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25153536, count: 4096 00000080:00200000:1.0:1789760761.124009:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25153536, bytes: 4096 00000080:00200000:1.0:1789760761.124023:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760761.124031:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760761.124034:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25153536, 25157632) 0 0 00000080:00200000:1.0:1789760761.124038:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760761.124056:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6141, 6141] 00020000:00200000:1.0:1789760761.124066:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25153536 chunk: [25153536, 25157632) 25157632, 4096 00020000:00200000:1.0:1789760761.124070:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760761.124092:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6141, 6141] 00000008:00010000:1.0:1789760761.124111:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760761.124117:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760761.124131:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25153536, 25157632) append 0 append lockpos 0 00000080:00200000:1.0:1789760761.124146:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25153536=0x17fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760761.124264:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760761.124267:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25157632 00000020:00010000:1.0:1789760761.124276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760761.124280:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.124283:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.124287:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760761.124290:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760761.124298:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760761.124305:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760761.124325:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760761.124529:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 25153536<>25157632 00000100:00100000:0.0:1789760761.124615:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495867904:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760761.166325:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973458 gen 2 00000008:00010000:1.0:1789760761.166389:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760761.166698:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.166710:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.166737:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760761.166771:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495867904:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.166911:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6141, 6141] 00000080:00200000:2.0:1789760761.166928:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.166935:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.166951:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25157632 i_blocks 49159 00000020:00200000:2.0:1789760761.166955:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.166967:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.167002:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25157632, count: 4096 00000080:00200000:2.0:1789760761.167009:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25157632, bytes: 4096 00000080:00200000:2.0:1789760761.167020:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.167030:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.167032:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25157632, 25161728) 0 0 00000080:00200000:2.0:1789760761.167037:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.167056:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6142, 6142] 00020000:00200000:2.0:1789760761.167065:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25157632 chunk: [25157632, 25161728) 25161728, 4096 00020000:00200000:2.0:1789760761.167070:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.167091:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6142, 6142] 00000008:00010000:2.0:1789760761.167109:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.167115:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.167127:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25157632, 25161728) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.167143:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25157632=0x17fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.172968:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.172981:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25161728 00000020:00010000:2.0:1789760761.172999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.173004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.173009:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.173013:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.173016:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.173031:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.173041:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.173078:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760761.177013:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 25157632<>25161728 00000100:00100000:3.0:1789760761.178906:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760761.178946:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868032:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760761.224410:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760761.224427:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.224450:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.224479:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760761.224510:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868032:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.224528:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6142, 6142] 00000080:00200000:2.0:1789760761.224550:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.224559:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.224575:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25161728 i_blocks 49159 00000020:00200000:2.0:1789760761.224579:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.224589:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.224624:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25161728, count: 4096 00000080:00200000:2.0:1789760761.224632:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25161728, bytes: 4096 00000080:00200000:2.0:1789760761.224654:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.224664:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.224667:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25161728, 25165824) 0 0 00000080:00200000:2.0:1789760761.224672:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.224690:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6143, 6143] 00020000:00200000:2.0:1789760761.224699:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25161728 chunk: [25161728, 25165824) 25165824, 4096 00020000:00200000:2.0:1789760761.224704:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.224727:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6143, 6143] 00000008:00010000:2.0:1789760761.224745:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.224752:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.224764:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25161728, 25165824) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.224780:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25161728=0x17ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.224841:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.224845:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25165824 00000020:00010000:2.0:1789760761.224863:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.224868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.224871:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.224875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.224878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.224886:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.224893:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.224909:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760761.225053:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 25161728<>25165824 00000100:00100000:3.0:1789760761.226766:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868160:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.278148:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973460 gen 2 00000008:00010000:3.0:1789760761.278245:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760761.278264:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.278275:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.278302:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760761.278332:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868160:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.279026:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6143, 6143] 00000080:00200000:2.0:1789760761.279057:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.279065:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.279081:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25165824 i_blocks 49159 00000020:00200000:2.0:1789760761.279085:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.279096:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.279130:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25165824, count: 4096 00000080:00200000:2.0:1789760761.279137:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25165824, bytes: 4096 00000080:00200000:2.0:1789760761.279149:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.279157:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.279160:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25165824, 25169920) 0 0 00000080:00200000:2.0:1789760761.279164:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.279207:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6144, 6144] 00020000:00200000:2.0:1789760761.279216:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25165824 chunk: [25165824, 25169920) 25169920, 4096 00020000:00200000:2.0:1789760761.279221:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.279240:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6144, 6144] 00000008:00010000:2.0:1789760761.279259:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.279265:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.279276:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25165824, 25169920) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.279289:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25165824=0x1800000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.279344:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.279347:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25169920 00000020:00010000:2.0:1789760761.279355:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.279359:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.279363:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.279366:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.279369:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.279376:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.279383:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.279401:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760761.279534:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 25165824<>25169920 00000100:00100000:3.0:1789760761.279612:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760761.279633:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868288:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760761.293081:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760761.336394:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973461 gen 2 00000008:00010000:3.0:1789760761.336450:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760761.336466:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.336479:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.336504:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760761.336537:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868288:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.336550:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6144, 6144] 00000080:00200000:2.0:1789760761.336572:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.336579:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.336597:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25169920 i_blocks 49159 00000020:00200000:2.0:1789760761.336600:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.336610:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.336645:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25169920, count: 4096 00000080:00200000:2.0:1789760761.336653:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25169920, bytes: 4096 00000080:00200000:2.0:1789760761.336662:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.336672:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.336675:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25169920, 25174016) 0 0 00000080:00200000:2.0:1789760761.336680:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.336697:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6145, 6145] 00020000:00200000:2.0:1789760761.336706:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25169920 chunk: [25169920, 25174016) 25174016, 4096 00020000:00200000:2.0:1789760761.336711:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.336734:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6145, 6145] 00000008:00010000:2.0:1789760761.336754:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.336759:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.336770:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25169920, 25174016) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.336784:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25169920=0x1801000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.336843:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.336846:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25174016 00000020:00010000:2.0:1789760761.336855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.336859:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.336862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.336865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.336868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.336875:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.336882:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.336898:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760761.337023:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 25169920<>25174016 00000100:00100000:3.0:1789760761.337209:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.363810:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973462 gen 2 00000008:00010000:3.0:1789760761.363864:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760761.363873:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.363885:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.363909:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760761.363940:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868416:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.364211:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6145, 6145] 00000080:00200000:2.0:1789760761.364230:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.364237:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.364252:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25174016 i_blocks 51209 00000020:00200000:2.0:1789760761.364256:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.364265:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.364298:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25174016, count: 4096 00000080:00200000:2.0:1789760761.364305:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25174016, bytes: 4096 00000080:00200000:2.0:1789760761.364315:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.364325:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.364328:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25174016, 25178112) 0 0 00000080:00200000:2.0:1789760761.364332:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.364348:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6146, 6146] 00020000:00200000:2.0:1789760761.364358:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25174016 chunk: [25174016, 25178112) 25178112, 4096 00020000:00200000:2.0:1789760761.364363:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.364383:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6146, 6146] 00000008:00010000:2.0:1789760761.364400:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.364406:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.364417:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25174016, 25178112) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.364431:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25174016=0x1802000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.364504:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.364507:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25178112 00000020:00010000:2.0:1789760761.364523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.364527:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.364531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.364535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.364538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.364545:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.364552:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.364567:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760761.364686:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 25174016<>25178112 00000100:00100000:3.0:1789760761.364919:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760761.364938:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.403142:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973463 gen 2 00000008:00010000:3.0:1789760761.403213:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760761.403222:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.403235:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.403265:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760761.403300:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868544:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.403814:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6146, 6146] 00000080:00200000:2.0:1789760761.403835:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.403842:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.403862:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25178112 i_blocks 51209 00000020:00200000:2.0:1789760761.403866:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.403879:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.403918:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25178112, count: 4096 00000080:00200000:2.0:1789760761.403926:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25178112, bytes: 4096 00000080:00200000:2.0:1789760761.403943:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.403952:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.403954:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25178112, 25182208) 0 0 00000080:00200000:2.0:1789760761.403959:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.403977:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6147, 6147] 00020000:00200000:2.0:1789760761.403986:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25178112 chunk: [25178112, 25182208) 25182208, 4096 00020000:00200000:2.0:1789760761.403991:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.404014:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6147, 6147] 00000008:00010000:2.0:1789760761.404033:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.404040:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.404054:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25178112, 25182208) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.404071:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25178112=0x1803000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.404129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.404133:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25182208 00000020:00010000:2.0:1789760761.404141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.404144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.404147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.404151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.404154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.404161:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.405626:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.405804:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:3.0:1789760761.405959:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 25178112<>25182208 00000100:00100000:3.0:1789760761.406054:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868672:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.447614:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973464 gen 2 00000008:00010000:3.0:1789760761.447666:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760761.447675:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.447688:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.447715:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760761.447747:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868672:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.447886:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6147, 6147] 00000080:00200000:2.0:1789760761.447906:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.447911:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.447926:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25182208 i_blocks 51209 00000020:00200000:2.0:1789760761.447930:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.447939:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.447974:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25182208, count: 4096 00000080:00200000:2.0:1789760761.447983:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25182208, bytes: 4096 00000080:00200000:2.0:1789760761.447993:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.448001:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.448004:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25182208, 25186304) 0 0 00000080:00200000:2.0:1789760761.448008:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.448025:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6148, 6148] 00020000:00200000:2.0:1789760761.448033:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25182208 chunk: [25182208, 25186304) 25186304, 4096 00020000:00200000:2.0:1789760761.448037:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.448058:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6148, 6148] 00000008:00010000:2.0:1789760761.448078:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.448084:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.448096:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25182208, 25186304) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.448111:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25182208=0x1804000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.448162:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.448164:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25186304 00000020:00010000:2.0:1789760761.448205:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.448209:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.448212:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.448216:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.448219:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.448226:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.448233:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.448364:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:3.0:1789760761.448501:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 25182208<>25186304 00000100:00100000:3.0:1789760761.448540:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760761.448555:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868800:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.486147:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973465 gen 2 00000008:00010000:3.0:1789760761.486229:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760761.486238:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.486249:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.486274:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760761.486304:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495868800:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.486613:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6148, 6148] 00000080:00200000:2.0:1789760761.486631:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.486637:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.486653:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25186304 i_blocks 51209 00000020:00200000:2.0:1789760761.486657:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.486666:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.486702:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25186304, count: 4096 00000080:00200000:2.0:1789760761.486708:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25186304, bytes: 4096 00000080:00200000:2.0:1789760761.486720:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.486729:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.486732:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25186304, 25190400) 0 0 00000080:00200000:2.0:1789760761.486736:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.486753:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6149, 6149] 00020000:00200000:2.0:1789760761.486761:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25186304 chunk: [25186304, 25190400) 25190400, 4096 00020000:00200000:2.0:1789760761.486766:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.486859:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6149, 6149] 00000008:00010000:2.0:1789760761.486878:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.486883:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.486896:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25186304, 25190400) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.486911:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25186304=0x1805000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.486967:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.486970:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25190400 00000020:00010000:2.0:1789760761.486979:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.486983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.486986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.486990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.486993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.487000:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.487007:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.487023:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760761.487144:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 25186304<>25190400 00000100:00100000:2.0:1789760761.487231:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.487251:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495868928:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760761.514498:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973466 gen 2 00000008:00010000:2.0:1789760761.514547:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760761.514557:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.514570:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.514594:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760761.514627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495868928:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760761.514967:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6149, 6149] 00000080:00200000:1.0:1789760761.514985:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760761.514991:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760761.515006:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25190400 i_blocks 51209 00000020:00200000:1.0:1789760761.515010:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760761.515019:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760761.515057:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25190400, count: 4096 00000080:00200000:1.0:1789760761.515064:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25190400, bytes: 4096 00000080:00200000:1.0:1789760761.515077:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760761.515086:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760761.515088:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25190400, 25194496) 0 0 00000080:00200000:1.0:1789760761.515093:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760761.515110:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6150, 6150] 00020000:00200000:1.0:1789760761.515119:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25190400 chunk: [25190400, 25194496) 25194496, 4096 00020000:00200000:1.0:1789760761.515123:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760761.515236:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6150, 6150] 00000008:00010000:1.0:1789760761.515258:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760761.515264:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760761.515276:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25190400, 25194496) append 0 append lockpos 0 00000080:00200000:1.0:1789760761.515291:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25190400=0x1806000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760761.515352:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760761.515355:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25194496 00000020:00010000:1.0:1789760761.515365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760761.515369:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.515372:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.515375:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760761.515378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760761.515385:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760761.515393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760761.515411:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760761.515558:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 25190400<>25194496 00000100:00100000:1.0:1789760761.515608:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760761.515627:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495869056:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760761.565366:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973467 gen 2 00000008:00010000:0.0:1789760761.565422:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760761.565430:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.565452:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.565483:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000080:00200000:1.0:1789760761.565625:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6150, 6150] 00000080:00200000:1.0:1789760761.565646:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760761.565653:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760761.565668:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25194496 i_blocks 51209 00000020:00200000:1.0:1789760761.565671:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760761.565679:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760761.565714:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25194496, count: 4096 00000080:00200000:1.0:1789760761.565722:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25194496, bytes: 4096 00000080:00200000:1.0:1789760761.565731:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760761.565740:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760761.565743:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25194496, 25198592) 0 0 00000080:00200000:1.0:1789760761.565747:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760761.565762:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6151, 6151] 00020000:00200000:1.0:1789760761.565770:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25194496 chunk: [25194496, 25198592) 25198592, 4096 00020000:00200000:1.0:1789760761.565774:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760761.566039:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6151, 6151] 00000008:00010000:1.0:1789760761.566058:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760761.566064:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760761.566076:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25194496, 25198592) append 0 append lockpos 0 00000080:00200000:1.0:1789760761.566089:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25194496=0x1807000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760761.566150:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760761.566153:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25198592 00000020:00010000:1.0:1789760761.566162:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760761.566165:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.566195:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.566199:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760761.566202:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760761.566211:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760761.566218:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760761.566239:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760761.566362:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 25194496<>25198592 00000100:00100000:1.0:1789760761.566426:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760761.566624:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495869056:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760761.566752:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495869184:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760761.618559:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973468 gen 2 00000008:00010000:0.0:1789760761.618615:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760761.618624:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.618637:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.618658:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000080:00200000:1.0:1789760761.618683:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6151, 6151] 00000100:00100000:0.0:1789760761.618686:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495869184:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760761.618702:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760761.618709:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760761.618725:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25198592 i_blocks 51209 00000020:00200000:1.0:1789760761.618729:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760761.618738:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760761.618773:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25198592, count: 4096 00000080:00200000:1.0:1789760761.619246:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25198592, bytes: 4096 00000080:00200000:1.0:1789760761.619261:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760761.619271:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760761.619273:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25198592, 25202688) 0 0 00000080:00200000:1.0:1789760761.619279:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760761.619297:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6152, 6152] 00020000:00200000:1.0:1789760761.619305:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25198592 chunk: [25198592, 25202688) 25202688, 4096 00020000:00200000:1.0:1789760761.619310:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760761.619330:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6152, 6152] 00000008:00010000:1.0:1789760761.619348:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760761.619355:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760761.619368:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25198592, 25202688) append 0 append lockpos 0 00000080:00200000:1.0:1789760761.619388:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25198592=0x1808000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760761.619450:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760761.619453:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25202688 00000020:00010000:1.0:1789760761.619471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760761.619475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.619478:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.619482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760761.619485:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760761.619687:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760761.619695:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760761.619716:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760761.619945:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 25198592<>25202688 00000100:00100000:1.0:1789760761.620290:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760761.620313:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495869312:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760761.682942:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973469 gen 2 00000008:00010000:0.0:1789760761.683068:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760761.683077:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.683088:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.683808:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760761.683841:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495869312:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.685799:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6152, 6152] 00000080:00200000:2.0:1789760761.685826:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.685834:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.685852:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25202688 i_blocks 51209 00000020:00200000:2.0:1789760761.685857:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.685873:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.685909:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25202688, count: 4096 00000080:00200000:2.0:1789760761.685918:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25202688, bytes: 4096 00000080:00200000:2.0:1789760761.686051:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.686062:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.686065:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25202688, 25206784) 0 0 00000080:00200000:2.0:1789760761.686070:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.686087:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6153, 6153] 00020000:00200000:2.0:1789760761.686095:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25202688 chunk: [25202688, 25206784) 25206784, 4096 00020000:00200000:2.0:1789760761.686099:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.686122:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6153, 6153] 00000008:00010000:2.0:1789760761.686143:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.686150:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.686162:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25202688, 25206784) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.686197:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25202688=0x1809000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.686269:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.686272:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25206784 00000020:00010000:2.0:1789760761.686282:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.686286:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.686289:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.686292:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.686295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.686302:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.686309:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.686338:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760761.686552:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 25202688<>25206784 00000100:00100000:3.0:1789760761.686662:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495869440:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.723651:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973470 gen 2 00000008:00010000:3.0:1789760761.723706:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760761.723714:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.723726:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.723930:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760761.723960:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495869440:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.726829:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6153, 6153] 00000080:00200000:2.0:1789760761.726857:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.726864:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.726883:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25206784 i_blocks 51209 00000020:00200000:2.0:1789760761.726888:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.726903:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.726938:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25206784, count: 4096 00000080:00200000:2.0:1789760761.726946:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25206784, bytes: 4096 00000080:00200000:2.0:1789760761.726961:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.726970:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.726973:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25206784, 25210880) 0 0 00000080:00200000:2.0:1789760761.726977:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.726996:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6154, 6154] 00020000:00200000:2.0:1789760761.727004:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25206784 chunk: [25206784, 25210880) 25210880, 4096 00020000:00200000:2.0:1789760761.727020:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.727783:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6154, 6154] 00000008:00010000:2.0:1789760761.727807:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.727812:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.727825:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25206784, 25210880) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.727841:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25206784=0x180a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.727909:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.727912:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25210880 00000020:00010000:2.0:1789760761.727920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.727924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.727927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.727931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.727934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.727942:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.727950:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.727983:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760761.728133:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 25206784<>25210880 00000100:00100000:3.0:1789760761.728240:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495869568:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760761.771824:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760761.771840:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.771854:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.771875:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760761.771911:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495869568:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.772097:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6154, 6154] 00000080:00200000:2.0:1789760761.772116:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.772122:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.772139:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25210880 i_blocks 51209 00000020:00200000:2.0:1789760761.772142:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.772152:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.772203:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25210880, count: 4096 00000080:00200000:2.0:1789760761.772211:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25210880, bytes: 4096 00000080:00200000:2.0:1789760761.772224:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.772234:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.772237:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25210880, 25214976) 0 0 00000080:00200000:2.0:1789760761.772242:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.772261:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6155, 6155] 00020000:00200000:2.0:1789760761.772270:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25210880 chunk: [25210880, 25214976) 25214976, 4096 00020000:00200000:2.0:1789760761.772274:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.772294:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6155, 6155] 00000008:00010000:2.0:1789760761.772312:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.772317:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.772329:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25210880, 25214976) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.772344:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25210880=0x180b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.772408:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.772411:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25214976 00000020:00010000:2.0:1789760761.772419:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.772423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.772426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.772429:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.772433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.772440:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.772448:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.772555:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:3.0:1789760761.772690:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 25210880<>25214976 00000100:00100000:3.0:1789760761.772731:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760761.772827:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495869696:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.816427:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973471 gen 2 00000100:00100000:3.0:1789760761.816634:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203144380 x1876699495869696/t4294973472(4294973472) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760777 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760761.816680:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760761.816686:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.816695:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.816716:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760761.816737:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495869696:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.817064:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6155, 6155] 00000080:00200000:2.0:1789760761.817082:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.817088:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.817107:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25214976 i_blocks 51209 00000020:00200000:2.0:1789760761.817112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.817121:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.817156:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25214976, count: 4096 00000080:00200000:2.0:1789760761.817162:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25214976, bytes: 4096 00000080:00200000:2.0:1789760761.817200:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.817209:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.817211:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25214976, 25219072) 0 0 00000080:00200000:2.0:1789760761.817215:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.817232:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6156, 6156] 00020000:00200000:2.0:1789760761.817240:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25214976 chunk: [25214976, 25219072) 25219072, 4096 00020000:00200000:2.0:1789760761.817244:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.817266:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6156, 6156] 00000008:00010000:2.0:1789760761.817289:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.817295:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.817308:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25214976, 25219072) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.817322:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25214976=0x180c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.817566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.817569:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25219072 00000020:00010000:2.0:1789760761.817579:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.817582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.817585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.817589:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.817592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.817599:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.817607:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.817626:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760761.817933:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 25214976<>25219072 00000100:00100000:2.0:1789760761.817988:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.818203:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495869824:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760761.865972:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973472 gen 2 00000100:00100000:2.0:1789760761.866194:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266680 x1876699495869824/t4294973473(4294973473) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760777 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760761.866248:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760761.866255:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.866267:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.866301:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760761.866332:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495869824:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760761.866494:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6156, 6156] 00000080:00200000:3.0:1789760761.866515:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.866521:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.866537:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25219072 i_blocks 51209 00000020:00200000:3.0:1789760761.866541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.866552:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.866588:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25219072, count: 4096 00000080:00200000:3.0:1789760761.866595:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25219072, bytes: 4096 00000080:00200000:3.0:1789760761.866611:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.866620:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.866622:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25219072, 25223168) 0 0 00000080:00200000:3.0:1789760761.866627:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.866646:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6157, 6157] 00020000:00200000:3.0:1789760761.866655:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25219072 chunk: [25219072, 25223168) 25223168, 4096 00020000:00200000:3.0:1789760761.866660:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.866682:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6157, 6157] 00000008:00010000:3.0:1789760761.866700:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.866706:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760761.866718:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25219072, 25223168) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.866733:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25219072=0x180d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.867156:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.867159:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25223168 00000020:00010000:3.0:1789760761.867388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.867392:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.867396:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.867400:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.867403:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.867415:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.867424:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.867444:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760761.868984:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 25219072<>25223168 00000100:00100000:3.0:1789760761.869544:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760761.869573:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495869952:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.910771:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973474 gen 2 00000008:00010000:3.0:1789760761.910936:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:3.0:1789760761.910945:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.910955:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.910980:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:3.0:1789760761.911010:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495869952:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.911118:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6157, 6157] 00000080:00200000:2.0:1789760761.911284:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.911291:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.911307:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25223168 i_blocks 51209 00000020:00200000:2.0:1789760761.911310:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.911320:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.911421:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25223168, count: 4096 00000080:00200000:2.0:1789760761.911430:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25223168, bytes: 4096 00000080:00200000:2.0:1789760761.911439:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.911448:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.911451:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25223168, 25227264) 0 0 00000080:00200000:2.0:1789760761.911464:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.911482:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6158, 6158] 00020000:00200000:2.0:1789760761.911490:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25223168 chunk: [25223168, 25227264) 25227264, 4096 00020000:00200000:2.0:1789760761.911495:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.911514:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6158, 6158] 00000008:00010000:2.0:1789760761.911533:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.911538:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.911550:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25223168, 25227264) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.911564:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25223168=0x180e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.911622:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.911626:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25227264 00000020:00010000:2.0:1789760761.911633:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.911637:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.911640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.911644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.911647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.911655:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.911662:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.911681:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760761.911809:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 25223168<>25227264 00000100:00100000:3.0:1789760761.911961:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760761.956336:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973475 gen 2 00000008:00010000:3.0:1789760761.956384:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760761.956393:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.956404:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.956427:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760761.956473:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870080:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760761.956613:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6158, 6158] 00000080:00200000:2.0:1789760761.956631:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760761.956637:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760761.956653:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25227264 i_blocks 51209 00000020:00200000:2.0:1789760761.956657:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760761.956666:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760761.956699:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25227264, count: 4096 00000080:00200000:2.0:1789760761.956706:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25227264, bytes: 4096 00000080:00200000:2.0:1789760761.956714:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760761.956727:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760761.956730:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25227264, 25231360) 0 0 00000080:00200000:2.0:1789760761.956734:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760761.956753:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6159, 6159] 00020000:00200000:2.0:1789760761.956761:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25227264 chunk: [25227264, 25231360) 25231360, 4096 00020000:00200000:2.0:1789760761.956766:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760761.956786:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6159, 6159] 00000008:00010000:2.0:1789760761.956804:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760761.956810:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760761.956822:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25227264, 25231360) append 0 append lockpos 0 00000080:00200000:2.0:1789760761.956836:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25227264=0x180f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760761.957701:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760761.957708:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25231360 00000020:00010000:2.0:1789760761.957718:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760761.957723:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.957726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760761.957730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760761.957733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760761.957742:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760761.957751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760761.957774:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760761.958190:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 25227264<>25231360 00000100:00100000:3.0:1789760761.958281:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495870208:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760762.004715:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973476 gen 2 00000008:00010000:3.0:1789760762.004764:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760762.004772:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.004893:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.004916:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760762.004956:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495870208:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760762.010712:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6159, 6159] 00000080:00200000:2.0:1789760762.010744:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.010751:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.010770:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25231360 i_blocks 51209 00000020:00200000:2.0:1789760762.010776:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.010794:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.010831:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25231360, count: 4096 00000080:00200000:2.0:1789760762.010840:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25231360, bytes: 4096 00000080:00200000:2.0:1789760762.010856:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.010865:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.010867:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25231360, 25235456) 0 0 00000080:00200000:2.0:1789760762.010872:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.010890:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6160, 6160] 00020000:00200000:2.0:1789760762.010898:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25231360 chunk: [25231360, 25235456) 25235456, 4096 00020000:00200000:2.0:1789760762.010903:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.010927:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6160, 6160] 00000008:00010000:2.0:1789760762.010945:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.010951:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760762.011029:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25231360, 25235456) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.011048:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25231360=0x1810000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.011127:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.011130:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25235456 00000020:00010000:2.0:1789760762.011139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.011143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.011146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.011150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.011152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.011160:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.011495:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.011588:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760762.011729:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 25231360<>25235456 00000100:00100000:3.0:1789760762.012580:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870336:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760762.047675:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973477 gen 2 00000008:00010000:3.0:1789760762.047722:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760762.047730:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.047742:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.047902:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760762.047931:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870336:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760762.048279:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6160, 6160] 00000080:00200000:2.0:1789760762.048296:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.048303:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.048586:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25235456 i_blocks 51209 00000020:00200000:0.0:1789760762.048592:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.048603:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.048639:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25235456, count: 4096 00000080:00200000:0.0:1789760762.048647:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25235456, bytes: 4096 00000080:00200000:0.0:1789760762.048661:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.048671:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.048673:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25235456, 25239552) 0 0 00000080:00200000:0.0:1789760762.048678:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.048695:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6161, 6161] 00020000:00200000:0.0:1789760762.048705:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25235456 chunk: [25235456, 25239552) 25239552, 4096 00020000:00200000:0.0:1789760762.048710:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.048733:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6161, 6161] 00000008:00010000:0.0:1789760762.048754:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.048761:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760762.048900:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25235456, 25239552) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.048917:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25235456=0x1811000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.049050:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.049054:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25239552 00000020:00010000:0.0:1789760762.049065:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.049069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.049072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.049076:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.049080:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.049088:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.049096:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.049208:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760762.049346:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25235456<>25239552 00000100:00100000:1.0:1789760762.049569:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495870464:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760762.098274:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973478 gen 2 00000008:00010000:0.0:1789760762.098328:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760762.098337:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.098348:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.098380:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760762.098413:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495870464:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760762.098836:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6161, 6161] 00000080:00200000:2.0:1789760762.099051:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.099059:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.099075:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25239552 i_blocks 51209 00000020:00200000:2.0:1789760762.099079:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.099089:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.099123:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25239552, count: 4096 00000080:00200000:2.0:1789760762.099130:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25239552, bytes: 4096 00000080:00200000:2.0:1789760762.099144:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.099153:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.099156:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25239552, 25243648) 0 0 00000080:00200000:2.0:1789760762.099160:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.099223:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6162, 6162] 00020000:00200000:2.0:1789760762.099232:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25239552 chunk: [25239552, 25243648) 25243648, 4096 00020000:00200000:2.0:1789760762.099238:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.099259:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6162, 6162] 00000008:00010000:2.0:1789760762.099277:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.099283:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760762.099296:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25239552, 25243648) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.099311:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25239552=0x1812000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.099372:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.099375:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25243648 00000020:00010000:2.0:1789760762.099384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.099388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.099391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.099394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.099397:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.099405:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.099413:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.099565:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760762.099693:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 25239552<>25243648 00000100:00100000:2.0:1789760762.099932:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760762.099955:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870592:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760762.140931:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973479 gen 2 00000008:00010000:2.0:1789760762.140980:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760762.140989:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.141000:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.141021:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760762.141048:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870592:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760762.141375:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6162, 6162] 00000080:00200000:3.0:1789760762.141392:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.141398:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.141413:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25243648 i_blocks 51209 00000020:00200000:3.0:1789760762.141417:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.141427:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.141534:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25243648, count: 4096 00000080:00200000:3.0:1789760762.141542:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25243648, bytes: 4096 00000080:00200000:3.0:1789760762.141553:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.141562:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.141565:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25243648, 25247744) 0 0 00000080:00200000:3.0:1789760762.141569:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.141586:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6163, 6163] 00020000:00200000:3.0:1789760762.141594:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25243648 chunk: [25243648, 25247744) 25247744, 4096 00020000:00200000:3.0:1789760762.141599:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.141619:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6163, 6163] 00000008:00010000:3.0:1789760762.141638:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.141644:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760762.141656:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25243648, 25247744) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.141669:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25243648=0x1813000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.141726:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.141729:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25247744 00000020:00010000:3.0:1789760762.141738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.141741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.141744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.141748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.141751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.141759:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.141765:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.141843:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760762.141978:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 25243648<>25247744 00000100:00100000:2.0:1789760762.142045:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870720:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760762.170728:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973480 gen 2 00000008:00010000:2.0:1789760762.170780:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:2.0:1789760762.170788:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.170799:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.170820:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:2.0:1789760762.170859:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870720:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760762.171015:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6163, 6163] 00000080:00200000:3.0:1789760762.171032:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.171038:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.171054:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25247744 i_blocks 51209 00000020:00200000:3.0:1789760762.171058:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.171067:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.171103:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25247744, count: 4096 00000080:00200000:3.0:1789760762.171110:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25247744, bytes: 4096 00000080:00200000:3.0:1789760762.171120:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.171128:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.171131:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25247744, 25251840) 0 0 00000080:00200000:3.0:1789760762.171135:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.171152:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6164, 6164] 00020000:00200000:3.0:1789760762.171161:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25247744 chunk: [25247744, 25251840) 25251840, 4096 00020000:00200000:3.0:1789760762.171165:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.171231:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6164, 6164] 00000008:00010000:3.0:1789760762.171249:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.171254:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760762.171268:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25247744, 25251840) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.171282:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25247744=0x1814000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.171335:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.171338:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25251840 00000020:00010000:3.0:1789760762.171348:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.171351:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.171355:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.171358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.171361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.171370:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.171376:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.171392:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760762.171589:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 25247744<>25251840 00000100:00100000:2.0:1789760762.172026:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495870848:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760762.200913:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973481 gen 2 00000008:00010000:2.0:1789760762.200967:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760762.200976:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.200986:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.201010:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:2.0:1789760762.201039:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495870848:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760762.201217:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6164, 6164] 00000080:00200000:3.0:1789760762.201235:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.201242:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.201258:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25251840 i_blocks 51209 00000020:00200000:3.0:1789760762.201261:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.201271:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.201304:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25251840, count: 4096 00000080:00200000:3.0:1789760762.201311:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25251840, bytes: 4096 00000080:00200000:3.0:1789760762.201321:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.201329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.201331:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25251840, 25255936) 0 0 00000080:00200000:3.0:1789760762.201336:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.201355:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6165, 6165] 00020000:00200000:3.0:1789760762.201364:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25251840 chunk: [25251840, 25255936) 25255936, 4096 00020000:00200000:3.0:1789760762.201369:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.201389:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6165, 6165] 00000008:00010000:3.0:1789760762.201408:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.201414:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760762.201427:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25251840, 25255936) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.201441:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25251840=0x1815000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.201517:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.201520:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25255936 00000020:00010000:3.0:1789760762.201530:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.201533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.201536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.201540:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.201543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.201551:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.201558:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.201575:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760762.201700:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 25251840<>25255936 00000100:00100000:2.0:1789760762.201986:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870976:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760762.254419:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973482 gen 2 00000008:00010000:2.0:1789760762.254482:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760762.254491:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.254503:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.254529:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760762.254559:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495870976:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760762.254657:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6165, 6165] 00000080:00200000:3.0:1789760762.254675:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.254682:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.254698:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25255936 i_blocks 51209 00000020:00200000:3.0:1789760762.254702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.254712:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.254745:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25255936, count: 4096 00000080:00200000:3.0:1789760762.254753:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25255936, bytes: 4096 00000080:00200000:3.0:1789760762.254763:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.254772:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.254786:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25255936, 25260032) 0 0 00000080:00200000:3.0:1789760762.254790:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.254807:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6166, 6166] 00020000:00200000:3.0:1789760762.254815:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25255936 chunk: [25255936, 25260032) 25260032, 4096 00020000:00200000:3.0:1789760762.254819:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.254839:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6166, 6166] 00000008:00010000:3.0:1789760762.254857:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.254863:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760762.254875:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25255936, 25260032) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.254888:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25255936=0x1816000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.254944:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.254948:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25260032 00000020:00010000:3.0:1789760762.254957:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.254961:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.254964:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.254967:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.254970:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.254989:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.254996:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.255011:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760762.255125:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 25255936<>25260032 00000100:00100000:2.0:1789760762.255592:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495871104:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760762.307847:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973483 gen 2 00000008:00010000:2.0:1789760762.307965:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760762.307985:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.307998:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.308021:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760762.308053:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495871104:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760762.308231:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6166, 6166] 00000080:00200000:3.0:1789760762.308247:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.308253:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.308276:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25260032 i_blocks 51209 00000020:00200000:3.0:1789760762.308280:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.308289:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.308325:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25260032, count: 4096 00000080:00200000:3.0:1789760762.308332:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25260032, bytes: 4096 00000080:00200000:3.0:1789760762.308343:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.308352:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.308354:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25260032, 25264128) 0 0 00000080:00200000:3.0:1789760762.308359:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.308377:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6167, 6167] 00020000:00200000:3.0:1789760762.308393:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25260032 chunk: [25260032, 25264128) 25264128, 4096 00020000:00200000:3.0:1789760762.308401:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.308421:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6167, 6167] 00000008:00010000:3.0:1789760762.308443:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.308450:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760762.308472:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25260032, 25264128) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.308486:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25260032=0x1817000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.308546:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.308549:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25264128 00000020:00010000:3.0:1789760762.308558:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.308562:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.308565:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.308569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.308572:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.308582:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.308590:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.308608:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760762.308752:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 25260032<>25264128 00000100:00100000:3.0:1789760762.308807:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760762.308823:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495871232:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760762.315995:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760762.366593:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973484 gen 2 00000008:00010000:3.0:1789760762.366651:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:3.0:1789760762.366659:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.366671:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.366704:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:3.0:1789760762.366736:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495871232:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760762.366745:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6167, 6167] 00000080:00200000:0.0:1789760762.366763:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.366769:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.366784:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25264128 i_blocks 51209 00000020:00200000:0.0:1789760762.366788:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.366798:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.366833:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25264128, count: 4096 00000080:00200000:0.0:1789760762.366840:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25264128, bytes: 4096 00000080:00200000:0.0:1789760762.366855:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.366865:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.366867:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25264128, 25268224) 0 0 00000080:00200000:0.0:1789760762.366871:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.366889:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6168, 6168] 00020000:00200000:0.0:1789760762.366898:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25264128 chunk: [25264128, 25268224) 25268224, 4096 00020000:00200000:0.0:1789760762.366903:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.366923:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6168, 6168] 00000008:00010000:0.0:1789760762.366941:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.366946:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760762.366958:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25264128, 25268224) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.366983:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25264128=0x1818000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.367575:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.367580:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25268224 00000020:00010000:0.0:1789760762.367590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.367593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.367597:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.367601:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.367604:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.367612:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.367620:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.367639:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760762.367774:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25264128<>25268224 00000100:00100000:0.0:1789760762.367831:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760762.367850:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495871360:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760762.409716:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760762.409734:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.409746:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.409818:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760762.409848:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495871360:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760762.409975:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6168, 6168] 00000080:00200000:1.0:1789760762.409992:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760762.409999:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760762.410015:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25268224 i_blocks 51209 00000020:00200000:1.0:1789760762.410106:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760762.410117:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760762.410152:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25268224, count: 4096 00000080:00200000:1.0:1789760762.410159:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25268224, bytes: 4096 00000080:00200000:1.0:1789760762.410396:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760762.410407:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760762.410410:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25268224, 25272320) 0 0 00000080:00200000:1.0:1789760762.410415:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760762.410433:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6169, 6169] 00020000:00200000:1.0:1789760762.410567:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25268224 chunk: [25268224, 25272320) 25272320, 4096 00020000:00200000:1.0:1789760762.410572:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760762.410595:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6169, 6169] 00000008:00010000:1.0:1789760762.410615:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760762.410620:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760762.410632:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25268224, 25272320) append 0 append lockpos 0 00000080:00200000:1.0:1789760762.410647:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25268224=0x1819000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760762.410711:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760762.410714:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25272320 00000020:00010000:1.0:1789760762.410725:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760762.410728:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.410731:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.410736:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760762.410740:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760762.410748:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760762.410755:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760762.410776:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760762.410971:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 25268224<>25272320 00000100:00100000:0.0:1789760762.411284:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495871488:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760762.452549:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973486 gen 2 00000008:00010000:0.0:1789760762.452737:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760762.452746:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.452756:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.452787:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760762.452827:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495871488:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760762.452930:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6169, 6169] 00000080:00200000:1.0:1789760762.452946:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760762.452953:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760762.452968:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25272320 i_blocks 51209 00000020:00200000:1.0:1789760762.452972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760762.452981:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760762.453014:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25272320, count: 4096 00000080:00200000:1.0:1789760762.453103:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25272320, bytes: 4096 00000080:00200000:1.0:1789760762.453114:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760762.453124:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760762.453126:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25272320, 25276416) 0 0 00000080:00200000:1.0:1789760762.453131:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760762.453147:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6170, 6170] 00020000:00200000:1.0:1789760762.453157:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25272320 chunk: [25272320, 25276416) 25276416, 4096 00020000:00200000:1.0:1789760762.453162:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760762.453206:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6170, 6170] 00000008:00010000:1.0:1789760762.453224:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760762.453230:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760762.453241:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25272320, 25276416) append 0 append lockpos 0 00000080:00200000:1.0:1789760762.453256:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25272320=0x181a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760762.453308:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760762.453311:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25276416 00000020:00010000:1.0:1789760762.453319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760762.453323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.453326:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.453331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760762.453334:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760762.453342:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760762.453349:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760762.453366:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760762.454157:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 25272320<>25276416 00000100:00100000:0.0:1789760762.454388:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495871616:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760762.487657:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973487 gen 2 00000008:00010000:0.0:1789760762.487754:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760762.487761:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.487771:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.487828:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760762.487853:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495871616:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760762.488250:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6170, 6170] 00000080:00200000:1.0:1789760762.488263:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760762.488267:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760762.488279:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25276416 i_blocks 51209 00000020:00200000:1.0:1789760762.488283:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760762.488291:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760762.488392:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25276416, count: 4096 00000080:00200000:1.0:1789760762.488399:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25276416, bytes: 4096 00000080:00200000:1.0:1789760762.488408:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760762.488415:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760762.488418:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25276416, 25280512) 0 0 00000080:00200000:1.0:1789760762.488422:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760762.488435:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6171, 6171] 00020000:00200000:1.0:1789760762.488443:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25276416 chunk: [25276416, 25280512) 25280512, 4096 00020000:00200000:1.0:1789760762.488448:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760762.488473:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6171, 6171] 00000008:00010000:1.0:1789760762.488487:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760762.488492:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760762.488501:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25276416, 25280512) append 0 append lockpos 0 00000080:00200000:1.0:1789760762.488511:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25276416=0x181b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760762.488557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760762.488560:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25280512 00000020:00010000:1.0:1789760762.488566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760762.488569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.488572:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.488576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760762.488578:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760762.488585:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760762.488591:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760762.488607:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760762.488766:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 25276416<>25280512 00000100:00100000:0.0:1789760762.488829:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495871744:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760762.540556:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973488 gen 2 00000008:00010000:0.0:1789760762.540595:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760762.540602:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.540611:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.540667:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760762.540694:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495871744:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760762.540770:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6171, 6171] 00000080:00200000:1.0:1789760762.540782:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760762.540787:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760762.540798:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25280512 i_blocks 51209 00000020:00200000:1.0:1789760762.540801:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760762.540810:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760762.540835:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25280512, count: 4096 00000080:00200000:1.0:1789760762.540841:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25280512, bytes: 4096 00000080:00200000:1.0:1789760762.540848:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760762.540854:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760762.540856:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25280512, 25284608) 0 0 00000080:00200000:1.0:1789760762.540860:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760762.540873:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6172, 6172] 00020000:00200000:1.0:1789760762.540880:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25280512 chunk: [25280512, 25284608) 25284608, 4096 00020000:00200000:1.0:1789760762.540885:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760762.540899:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6172, 6172] 00000008:00010000:1.0:1789760762.540914:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760762.540920:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760762.540928:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25280512, 25284608) append 0 append lockpos 0 00000080:00200000:1.0:1789760762.540937:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25280512=0x181c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760762.540981:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760762.540984:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25284608 00000020:00010000:1.0:1789760762.541131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760762.541136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.541139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.541143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760762.541146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760762.541152:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760762.541158:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760762.541538:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760762.541632:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 25280512<>25284608 00000100:00100000:1.0:1789760762.541675:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495871872:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760762.598609:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973489 gen 2 00000008:00010000:0.0:1789760762.598648:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760762.598654:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.598664:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.598684:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760762.598711:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495871872:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760762.598748:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6172, 6172] 00000080:00200000:3.0:1789760762.598839:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.598848:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.598875:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25284608 i_blocks 51209 00000020:00200000:3.0:1789760762.598879:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.598893:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.598928:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25284608, count: 4096 00000080:00200000:3.0:1789760762.598937:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25284608, bytes: 4096 00000080:00200000:3.0:1789760762.598949:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.598959:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.598961:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25284608, 25288704) 0 0 00000080:00200000:3.0:1789760762.598966:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.598983:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6173, 6173] 00020000:00200000:3.0:1789760762.598992:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25284608 chunk: [25284608, 25288704) 25288704, 4096 00020000:00200000:3.0:1789760762.598997:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.599017:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6173, 6173] 00000008:00010000:3.0:1789760762.599034:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.599040:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760762.599052:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25284608, 25288704) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.599069:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25284608=0x181d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.599129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.599132:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25288704 00000020:00010000:3.0:1789760762.599142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.599146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.599149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.599153:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.599156:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.599164:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.599217:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.599237:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760762.599424:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 25284608<>25288704 00000100:00100000:2.0:1789760762.599625:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495872000:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760762.633795:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973490 gen 2 00000008:00010000:2.0:1789760762.633848:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760762.633857:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.633868:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.633888:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760762.633917:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495872000:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760762.634067:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6173, 6173] 00000080:00200000:3.0:1789760762.634083:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.634090:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.634107:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25288704 i_blocks 51209 00000020:00200000:3.0:1789760762.634110:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.634119:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.634152:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25288704, count: 4096 00000080:00200000:3.0:1789760762.634159:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25288704, bytes: 4096 00000080:00200000:3.0:1789760762.634197:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.634206:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.634209:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25288704, 25292800) 0 0 00000080:00200000:3.0:1789760762.634213:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.634229:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6174, 6174] 00020000:00200000:3.0:1789760762.634238:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25288704 chunk: [25288704, 25292800) 25292800, 4096 00020000:00200000:3.0:1789760762.634242:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.634262:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6174, 6174] 00000008:00010000:3.0:1789760762.634279:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.634284:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760762.634296:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25288704, 25292800) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.634310:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25288704=0x181e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.634366:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.634369:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25292800 00000020:00010000:3.0:1789760762.634378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.634381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.634384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.634388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.634391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.634398:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.634406:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.634422:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760762.634568:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 25288704<>25292800 00000100:00100000:2.0:1789760762.635825:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495872128:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760762.666398:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973491 gen 2 00000008:00010000:2.0:1789760762.666462:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760762.666470:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.666483:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.666568:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760762.666601:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495872128:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760762.667420:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6174, 6174] 00000080:00200000:3.0:1789760762.667441:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.667459:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.667478:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25292800 i_blocks 51209 00000020:00200000:3.0:1789760762.667483:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.667494:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.667529:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25292800, count: 4096 00000080:00200000:3.0:1789760762.667538:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25292800, bytes: 4096 00000080:00200000:3.0:1789760762.667552:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.667561:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.667563:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25292800, 25296896) 0 0 00000080:00200000:3.0:1789760762.667569:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.667586:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6175, 6175] 00020000:00200000:3.0:1789760762.667595:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25292800 chunk: [25292800, 25296896) 25296896, 4096 00020000:00200000:3.0:1789760762.667600:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.667621:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6175, 6175] 00000008:00010000:3.0:1789760762.667641:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.667646:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760762.667659:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25292800, 25296896) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.667675:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25292800=0x181f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.667740:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.667743:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25296896 00000020:00010000:3.0:1789760762.667752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.667755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.667758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.667761:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.667764:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.671709:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.671731:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.672038:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760762.673888:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 25292800<>25296896 00000100:00100000:3.0:1789760762.673974:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760762.674002:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495872256:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760762.721094:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973492 gen 2 00000008:00010000:3.0:1789760762.721153:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760762.721162:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.721206:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.721229:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760762.721258:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495872256:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760762.721299:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6175, 6175] 00000080:00200000:2.0:1789760762.721317:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.721324:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.721340:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25296896 i_blocks 51209 00000020:00200000:2.0:1789760762.721344:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.721354:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.721389:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25296896, count: 4096 00000080:00200000:2.0:1789760762.721396:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25296896, bytes: 4096 00000080:00200000:2.0:1789760762.721406:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.721415:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.721417:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25296896, 25300992) 0 0 00000080:00200000:2.0:1789760762.721422:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.721459:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6176, 6176] 00020000:00200000:2.0:1789760762.721467:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25296896 chunk: [25296896, 25300992) 25300992, 4096 00020000:00200000:2.0:1789760762.721472:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.721492:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6176, 6176] 00000008:00010000:2.0:1789760762.721512:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.721519:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760762.721531:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25296896, 25300992) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.721546:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25296896=0x1820000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.721602:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.721605:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25300992 00000020:00010000:2.0:1789760762.721614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.721618:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.721621:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.721624:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.721627:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.721635:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.721642:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.721658:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760762.721778:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 25296896<>25300992 00000100:00100000:2.0:1789760762.721827:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760762.721844:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495872384:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760762.766245:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760762.766263:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.766277:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.766309:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760762.766341:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495872384:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760762.766352:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6176, 6176] 00000080:00200000:0.0:1789760762.766373:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.766380:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.766396:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25300992 i_blocks 51209 00000020:00200000:0.0:1789760762.766400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.766411:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.766510:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25300992, count: 4096 00000080:00200000:0.0:1789760762.766519:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25300992, bytes: 4096 00000080:00200000:0.0:1789760762.766531:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.766540:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.766542:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25300992, 25305088) 0 0 00000080:00200000:0.0:1789760762.766547:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.766564:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6177, 6177] 00020000:00200000:0.0:1789760762.766573:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25300992 chunk: [25300992, 25305088) 25305088, 4096 00020000:00200000:0.0:1789760762.766578:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.766598:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6177, 6177] 00000008:00010000:0.0:1789760762.766616:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.766622:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760762.766634:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25300992, 25305088) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.766648:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25300992=0x1821000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.766703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.766706:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25305088 00000020:00010000:0.0:1789760762.766715:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.766718:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.766721:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.766725:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.766729:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.766736:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.766744:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.766761:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760762.766885:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25300992<>25305088 00000100:00100000:0.0:1789760762.766932:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760762.766949:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495872512:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760762.804891:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973494 gen 2 00000008:00010000:0.0:1789760762.804955:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760762.804964:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.804973:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.805003:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760762.805034:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495872512:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760762.805194:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6177, 6177] 00000080:00200000:2.0:1789760762.805211:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.805217:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.805233:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25305088 i_blocks 51209 00000020:00200000:2.0:1789760762.805237:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.805246:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.805280:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25305088, count: 4096 00000080:00200000:2.0:1789760762.805288:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25305088, bytes: 4096 00000080:00200000:2.0:1789760762.805301:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.805310:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.805313:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25305088, 25309184) 0 0 00000080:00200000:2.0:1789760762.805317:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.805334:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6178, 6178] 00020000:00200000:2.0:1789760762.805342:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25305088 chunk: [25305088, 25309184) 25309184, 4096 00020000:00200000:2.0:1789760762.805347:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.805368:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6178, 6178] 00000008:00010000:2.0:1789760762.805386:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.805392:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760762.805404:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25305088, 25309184) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.805417:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25305088=0x1822000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.805480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.805483:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25309184 00000020:00010000:2.0:1789760762.805491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.805495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.805498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.805502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.805505:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.805512:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.805520:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.805536:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760762.805727:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 25305088<>25309184 00000100:00100000:3.0:1789760762.805806:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495872640:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760762.852844:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973495 gen 2 00000008:00010000:3.0:1789760762.852908:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760762.852917:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.852929:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.852954:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760762.852985:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495872640:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760762.853324:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6178, 6178] 00000080:00200000:2.0:1789760762.853342:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.853349:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.853366:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25309184 i_blocks 51209 00000020:00200000:2.0:1789760762.853370:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.853380:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.853415:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25309184, count: 4096 00000080:00200000:2.0:1789760762.853423:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25309184, bytes: 4096 00000080:00200000:2.0:1789760762.853436:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.853445:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.853457:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25309184, 25313280) 0 0 00000080:00200000:2.0:1789760762.853462:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.853480:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6179, 6179] 00020000:00200000:2.0:1789760762.853489:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25309184 chunk: [25309184, 25313280) 25313280, 4096 00020000:00200000:2.0:1789760762.853494:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.853515:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6179, 6179] 00000008:00010000:2.0:1789760762.853535:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.853541:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760762.853552:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25309184, 25313280) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.853568:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25309184=0x1823000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.853628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.853631:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25313280 00000020:00010000:2.0:1789760762.853640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.853644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.853648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.853651:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.853654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.853662:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.854011:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.854136:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760762.854321:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 25309184<>25313280 00000100:00100000:3.0:1789760762.854408:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495872768:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760762.896112:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973496 gen 2 00000008:00010000:3.0:1789760762.896163:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760762.896200:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.896212:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.896237:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760762.896266:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495872768:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760762.896384:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6179, 6179] 00000080:00200000:2.0:1789760762.896401:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.896409:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.896494:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25313280 i_blocks 51209 00000020:00200000:2.0:1789760762.896499:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.896508:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.896543:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25313280, count: 4096 00000080:00200000:2.0:1789760762.896550:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25313280, bytes: 4096 00000080:00200000:2.0:1789760762.896559:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.896567:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.896569:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25313280, 25317376) 0 0 00000080:00200000:2.0:1789760762.896573:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.896589:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6180, 6180] 00020000:00200000:2.0:1789760762.896597:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25313280 chunk: [25313280, 25317376) 25317376, 4096 00020000:00200000:2.0:1789760762.896601:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.896621:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6180, 6180] 00000008:00010000:2.0:1789760762.896639:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.896644:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760762.896656:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25313280, 25317376) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.896670:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25313280=0x1824000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.896733:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.896736:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25317376 00000020:00010000:2.0:1789760762.896745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.896748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.896751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.896754:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.896757:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.896765:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.897370:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.897392:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760762.897632:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 25313280<>25317376 00000100:00100000:3.0:1789760762.897706:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495872896:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760762.927921:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760762.927938:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.927950:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.927975:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000080:00200000:2.0:1789760762.928221:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6180, 6180] 00000080:00200000:2.0:1789760762.928240:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.928248:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.928264:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25317376 i_blocks 51209 00000020:00200000:2.0:1789760762.928268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.928278:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.928312:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25317376, count: 4096 00000080:00200000:2.0:1789760762.928321:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25317376, bytes: 4096 00000080:00200000:2.0:1789760762.928331:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.928340:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.928343:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25317376, 25321472) 0 0 00000080:00200000:2.0:1789760762.928348:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.928366:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6181, 6181] 00020000:00200000:2.0:1789760762.928374:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25317376 chunk: [25317376, 25321472) 25321472, 4096 00020000:00200000:2.0:1789760762.928379:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.928400:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6181, 6181] 00000008:00010000:2.0:1789760762.928420:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.928426:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760762.928693:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25317376, 25321472) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.928709:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25317376=0x1825000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.928752:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.928754:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25321472 00000020:00010000:2.0:1789760762.928762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.928765:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.928768:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.928772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.928775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.928783:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.928792:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760762.928969:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495872896:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760762.929124:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:3.0:1789760762.929276:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 25317376<>25321472 00000100:00100000:2.0:1789760762.929345:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873024:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760762.972886:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973498 gen 2 00000008:00010000:2.0:1789760762.972958:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760762.972968:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.972978:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.973009:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760762.973040:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873024:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760762.973160:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6181, 6181] 00000080:00200000:0.0:1789760762.973208:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.973215:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.973231:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25321472 i_blocks 51209 00000020:00200000:0.0:1789760762.973235:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.973244:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.973279:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25321472, count: 4096 00000080:00200000:0.0:1789760762.973286:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25321472, bytes: 4096 00000080:00200000:0.0:1789760762.973296:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.973306:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.973308:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25321472, 25325568) 0 0 00000080:00200000:0.0:1789760762.973312:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.973330:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6182, 6182] 00020000:00200000:0.0:1789760762.973339:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25321472 chunk: [25321472, 25325568) 25325568, 4096 00020000:00200000:0.0:1789760762.973344:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.973363:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6182, 6182] 00000008:00010000:0.0:1789760762.973380:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.973386:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760762.973398:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25321472, 25325568) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.973413:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25321472=0x1826000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.973670:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.973675:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25325568 00000020:00010000:0.0:1789760762.973685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.973689:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.973692:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.973695:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.973698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.973706:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.973714:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.973733:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760762.973861:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25321472<>25325568 00000100:00100000:1.0:1789760762.974621:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495873152:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760763.017721:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973499 gen 2 00000008:00010000:0.0:1789760763.017771:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760763.017781:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.017791:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.017879:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760763.017909:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495873152:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760763.019729:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6182, 6182] 00000080:00200000:2.0:1789760763.019762:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.019769:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.020080:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25325568 i_blocks 51209 00000020:00200000:2.0:1789760763.020153:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.020195:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.020237:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25325568, count: 4096 00000080:00200000:2.0:1789760763.020247:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25325568, bytes: 4096 00000080:00200000:2.0:1789760763.020264:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.020274:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.020277:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25325568, 25329664) 0 0 00000080:00200000:2.0:1789760763.020282:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.020302:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6183, 6183] 00020000:00200000:2.0:1789760763.020312:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25325568 chunk: [25325568, 25329664) 25329664, 4096 00020000:00200000:2.0:1789760763.020317:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.020349:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6183, 6183] 00000008:00010000:2.0:1789760763.020368:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.020375:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760763.020388:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25325568, 25329664) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.020407:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25325568=0x1827000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.021401:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.021408:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25329664 00000020:00010000:2.0:1789760763.021420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.021424:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.021428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.021432:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.021435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.021615:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.021626:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.021656:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760763.024285:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 25325568<>25329664 00000100:00100000:3.0:1789760763.024598:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495873280:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760763.083308:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973500 gen 2 00000008:00010000:3.0:1789760763.083361:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760763.083371:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.083382:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.083411:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760763.083458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495873280:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760763.083702:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6183, 6183] 00000080:00200000:0.0:1789760763.083719:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.083725:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.083741:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25329664 i_blocks 51209 00000020:00200000:0.0:1789760763.083745:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.083754:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.083798:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25329664, count: 4096 00000080:00200000:0.0:1789760763.083806:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25329664, bytes: 4096 00000080:00200000:0.0:1789760763.083817:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.083825:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.083828:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25329664, 25333760) 0 0 00000080:00200000:0.0:1789760763.083833:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.084140:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6184, 6184] 00020000:00200000:0.0:1789760763.084151:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25329664 chunk: [25329664, 25333760) 25333760, 4096 00020000:00200000:0.0:1789760763.084155:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.084205:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6184, 6184] 00000008:00010000:0.0:1789760763.084224:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.084230:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760763.084242:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25329664, 25333760) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.084257:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25329664=0x1828000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.084313:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.084315:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25333760 00000020:00010000:0.0:1789760763.084325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.084328:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.084331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.084335:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.084338:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.084345:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.084351:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.084368:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760763.084569:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25329664<>25333760 00000100:00100000:1.0:1789760763.084646:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495873408:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760763.133064:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973501 gen 2 00000008:00010000:0.0:1789760763.133117:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760763.133125:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.133138:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.133274:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760763.133303:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495873408:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.133780:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6184, 6184] 00000080:00200000:3.0:1789760763.133799:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.133806:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.133823:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25333760 i_blocks 51209 00000020:00200000:3.0:1789760763.133828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.133838:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.133873:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25333760, count: 4096 00000080:00200000:3.0:1789760763.133882:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25333760, bytes: 4096 00000080:00200000:3.0:1789760763.133895:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.133904:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.133906:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25333760, 25337856) 0 0 00000080:00200000:3.0:1789760763.133911:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.133929:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6185, 6185] 00020000:00200000:3.0:1789760763.133939:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25333760 chunk: [25333760, 25337856) 25337856, 4096 00020000:00200000:3.0:1789760763.133944:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.133964:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6185, 6185] 00000008:00010000:3.0:1789760763.133985:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.133992:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760763.134004:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25333760, 25337856) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.134019:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25333760=0x1829000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.134080:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.134083:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25337856 00000020:00010000:3.0:1789760763.134092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.134096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.134099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.134103:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.134107:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.134115:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.134122:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.134141:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760763.134336:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 25333760<>25337856 00000100:00100000:2.0:1789760763.134432:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873536:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760763.179403:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973502 gen 2 00000008:00010000:2.0:1789760763.179463:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760763.179472:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.179483:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.179502:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760763.179540:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873536:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.180579:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6185, 6185] 00000080:00200000:3.0:1789760763.180605:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.180613:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.180632:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25337856 i_blocks 51209 00000020:00200000:3.0:1789760763.180637:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.180648:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.180683:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25337856, count: 4096 00000080:00200000:3.0:1789760763.180690:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25337856, bytes: 4096 00000080:00200000:3.0:1789760763.180702:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.180713:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.180716:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25337856, 25341952) 0 0 00000080:00200000:3.0:1789760763.180730:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.180749:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6186, 6186] 00020000:00200000:3.0:1789760763.180757:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25337856 chunk: [25337856, 25341952) 25341952, 4096 00020000:00200000:3.0:1789760763.180762:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.180785:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6186, 6186] 00000008:00010000:3.0:1789760763.180803:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.180809:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760763.180826:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25337856, 25341952) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.180840:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25337856=0x182a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.180905:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.180908:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25341952 00000020:00010000:3.0:1789760763.180917:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.180921:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.180924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.180927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.180931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.180939:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.180945:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.180961:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760763.181087:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 25337856<>25341952 00000100:00100000:2.0:1789760763.182904:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760763.182941:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873664:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760763.219732:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973503 gen 2 00000008:00010000:2.0:1789760763.219787:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760763.219795:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.219806:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.219832:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:2.0:1789760763.219862:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873664:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.219992:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6186, 6186] 00000080:00200000:3.0:1789760763.220019:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.220026:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.220042:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25341952 i_blocks 51209 00000020:00200000:3.0:1789760763.220047:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.220057:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.220096:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25341952, count: 4096 00000080:00200000:3.0:1789760763.220103:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25341952, bytes: 4096 00000080:00200000:3.0:1789760763.220115:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.220125:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.220127:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25341952, 25346048) 0 0 00000080:00200000:3.0:1789760763.220132:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.220149:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6187, 6187] 00020000:00200000:3.0:1789760763.220157:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25341952 chunk: [25341952, 25346048) 25346048, 4096 00020000:00200000:3.0:1789760763.220163:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.220224:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6187, 6187] 00000008:00010000:3.0:1789760763.220258:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.220268:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760763.220280:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25341952, 25346048) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.220297:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25341952=0x182b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.220367:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.220370:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25346048 00000020:00010000:3.0:1789760763.220379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.220383:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.220386:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.220390:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.220392:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.220400:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.220407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.220432:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760763.220825:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 25341952<>25346048 00000100:00100000:2.0:1789760763.221257:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873792:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760763.256699:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760763.256716:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.256727:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.256750:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760763.256780:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873792:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.257007:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6187, 6187] 00000080:00200000:3.0:1789760763.257025:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.257032:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.257050:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25346048 i_blocks 51209 00000020:00200000:3.0:1789760763.257054:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.257064:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.257096:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25346048, count: 4096 00000080:00200000:3.0:1789760763.257103:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25346048, bytes: 4096 00000080:00200000:3.0:1789760763.257113:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.257122:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.257124:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25346048, 25350144) 0 0 00000080:00200000:3.0:1789760763.257129:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.257148:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6188, 6188] 00020000:00200000:3.0:1789760763.257158:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25346048 chunk: [25346048, 25350144) 25350144, 4096 00020000:00200000:3.0:1789760763.257162:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.257218:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6188, 6188] 00000008:00010000:3.0:1789760763.257237:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.257243:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760763.257255:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25346048, 25350144) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.257269:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25346048=0x182c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.257324:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.257327:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25350144 00000020:00010000:3.0:1789760763.257335:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.257339:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.257584:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.257588:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.257591:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.257600:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.257608:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.257626:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760763.257947:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 25346048<>25350144 00000100:00100000:2.0:1789760763.258226:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760763.258423:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873920:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760763.296482:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973505 gen 2 00000008:00010000:2.0:1789760763.296553:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:2.0:1789760763.296562:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.296572:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.296649:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:2.0:1789760763.296809:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495873920:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.296900:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6188, 6188] 00000080:00200000:3.0:1789760763.296917:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.296924:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.297025:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25350144 i_blocks 51209 00000020:00200000:2.0:1789760763.297030:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.297040:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.297074:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25350144, count: 4096 00000080:00200000:2.0:1789760763.297082:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25350144, bytes: 4096 00000080:00200000:2.0:1789760763.297095:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.297104:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.297106:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25350144, 25354240) 0 0 00000080:00200000:2.0:1789760763.297110:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.297127:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6189, 6189] 00020000:00200000:2.0:1789760763.297135:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25350144 chunk: [25350144, 25354240) 25354240, 4096 00020000:00200000:2.0:1789760763.297140:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.297162:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6189, 6189] 00000008:00010000:2.0:1789760763.297209:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.297215:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760763.297228:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25350144, 25354240) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.297243:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25350144=0x182d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.297298:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.297302:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25354240 00000020:00010000:2.0:1789760763.297311:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.297314:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.297318:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.297321:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.297324:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.297331:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.297338:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.297428:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760763.297562:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 25350144<>25354240 00000100:00100000:3.0:1789760763.297624:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495874048:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760763.339635:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760763.345726:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973506 gen 2 00000008:00010000:2.0:1789760763.345778:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760763.345787:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.345987:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.346012:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760763.346041:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495874048:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760763.346154:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6189, 6189] 00000080:00200000:0.0:1789760763.346192:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.346198:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.346214:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25354240 i_blocks 51209 00000020:00200000:0.0:1789760763.346217:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.346227:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.346262:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25354240, count: 4096 00000080:00200000:0.0:1789760763.346269:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25354240, bytes: 4096 00000080:00200000:0.0:1789760763.346280:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.346288:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.346291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25354240, 25358336) 0 0 00000080:00200000:0.0:1789760763.346296:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.346314:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6190, 6190] 00020000:00200000:0.0:1789760763.346321:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25354240 chunk: [25354240, 25358336) 25358336, 4096 00020000:00200000:0.0:1789760763.346327:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.346345:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6190, 6190] 00000008:00010000:0.0:1789760763.346433:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.346440:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760763.346459:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25354240, 25358336) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.346474:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25354240=0x182e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.346527:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.346530:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25358336 00000020:00010000:0.0:1789760763.346538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.346541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.346544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.346547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.346550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.346557:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.346563:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.346579:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760763.346702:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25354240<>25358336 00000100:00100000:0.0:1789760763.346754:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760763.346770:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495874176:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760763.383252:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973507 gen 2 00000008:00010000:0.0:1789760763.383301:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760763.383309:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.383320:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.383349:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760763.383451:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495874176:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760763.383725:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6190, 6190] 00000080:00200000:2.0:1789760763.383743:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.383749:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.383765:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25358336 i_blocks 51209 00000020:00200000:2.0:1789760763.383769:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.383778:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.383821:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25358336, count: 4096 00000080:00200000:2.0:1789760763.383828:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25358336, bytes: 4096 00000080:00200000:2.0:1789760763.383839:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.383847:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.383849:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25358336, 25362432) 0 0 00000080:00200000:2.0:1789760763.383855:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.383872:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6191, 6191] 00020000:00200000:2.0:1789760763.383880:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25358336 chunk: [25358336, 25362432) 25362432, 4096 00020000:00200000:2.0:1789760763.383885:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.383904:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6191, 6191] 00000008:00010000:2.0:1789760763.383921:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.383926:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760763.383938:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25358336, 25362432) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.383951:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25358336=0x182f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.384009:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.384012:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25362432 00000020:00010000:2.0:1789760763.384021:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.384024:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.384027:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.384031:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.384034:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.384040:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.384046:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.384060:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760763.384224:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 25358336<>25362432 00000100:00100000:2.0:1789760763.384272:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760763.384289:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495874304:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760763.426412:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973508 gen 2 00000008:00010000:2.0:1789760763.426468:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760763.426477:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.426487:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.426509:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760763.426536:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495874304:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.426649:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6191, 6191] 00000080:00200000:3.0:1789760763.426665:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.426671:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.426687:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25362432 i_blocks 51209 00000020:00200000:3.0:1789760763.426690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.426700:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.426760:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25362432, count: 4096 00000080:00200000:3.0:1789760763.426768:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25362432, bytes: 4096 00000080:00200000:3.0:1789760763.426777:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.426786:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.426788:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25362432, 25366528) 0 0 00000080:00200000:3.0:1789760763.426805:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.426823:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6192, 6192] 00020000:00200000:3.0:1789760763.426831:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25362432 chunk: [25362432, 25366528) 25366528, 4096 00020000:00200000:3.0:1789760763.426836:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.426855:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6192, 6192] 00000008:00010000:3.0:1789760763.426874:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.426879:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760763.426889:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25362432, 25366528) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.426904:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25362432=0x1830000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.426961:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.426965:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25366528 00000020:00010000:3.0:1789760763.426974:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.426977:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.426980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.426984:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.426987:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.426994:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.427001:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.427017:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760763.427135:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 25362432<>25366528 00000100:00100000:3.0:1789760763.427217:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760763.427233:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495874432:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760763.473659:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973509 gen 2 00000008:00010000:3.0:1789760763.473712:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:3.0:1789760763.473721:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.473731:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.473756:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:3.0:1789760763.473785:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495874432:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760763.473960:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6192, 6192] 00000080:00200000:0.0:1789760763.473977:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.473983:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.473998:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25366528 i_blocks 51209 00000020:00200000:0.0:1789760763.474002:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.474011:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.474043:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25366528, count: 4096 00000080:00200000:0.0:1789760763.474050:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25366528, bytes: 4096 00000080:00200000:0.0:1789760763.474061:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.474069:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.474072:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25366528, 25370624) 0 0 00000080:00200000:0.0:1789760763.474076:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.474093:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6193, 6193] 00020000:00200000:0.0:1789760763.474101:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25366528 chunk: [25366528, 25370624) 25370624, 4096 00020000:00200000:0.0:1789760763.474106:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.474126:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6193, 6193] 00000008:00010000:0.0:1789760763.474144:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.474150:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760763.474162:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25366528, 25370624) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.474626:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25366528=0x1831000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.474689:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.474692:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25370624 00000020:00010000:0.0:1789760763.474701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.474705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.474708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.474711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.474715:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.474723:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.474730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.474749:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760763.475258:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25366528<>25370624 00000100:00100000:1.0:1789760763.475548:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.475571:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495874560:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760763.510558:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760763.510575:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.510586:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.510607:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760763.510639:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495874560:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.510735:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6193, 6193] 00000080:00200000:3.0:1789760763.510754:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.510760:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.510775:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25370624 i_blocks 51209 00000020:00200000:3.0:1789760763.510779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.510789:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.511426:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25370624, count: 4096 00000080:00200000:3.0:1789760763.511437:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25370624, bytes: 4096 00000080:00200000:3.0:1789760763.511460:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.511470:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.511472:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25370624, 25374720) 0 0 00000080:00200000:3.0:1789760763.511477:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.511493:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6194, 6194] 00020000:00200000:3.0:1789760763.511501:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25370624 chunk: [25370624, 25374720) 25374720, 4096 00020000:00200000:3.0:1789760763.511506:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.511526:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6194, 6194] 00000008:00010000:3.0:1789760763.511545:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.511550:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760763.511562:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25370624, 25374720) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.511576:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25370624=0x1832000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.511639:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.511642:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25374720 00000020:00010000:3.0:1789760763.511650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.511654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.511657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.511660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.511663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.511671:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.511678:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.511701:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760763.514938:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 25370624<>25374720 00000100:00100000:3.0:1789760763.515060:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495874688:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760763.553875:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973511 gen 2 00000008:00010000:3.0:1789760763.553946:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:3.0:1789760763.553954:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.553965:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.553993:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:3.0:1789760763.554021:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495874688:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760763.554133:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6194, 6194] 00000080:00200000:2.0:1789760763.554149:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.554155:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.554203:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25374720 i_blocks 51209 00000020:00200000:2.0:1789760763.554206:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.554216:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.554249:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25374720, count: 4096 00000080:00200000:2.0:1789760763.554257:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25374720, bytes: 4096 00000080:00200000:2.0:1789760763.554267:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.554275:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.554278:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25374720, 25378816) 0 0 00000080:00200000:2.0:1789760763.554282:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.554300:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6195, 6195] 00020000:00200000:2.0:1789760763.554308:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25374720 chunk: [25374720, 25378816) 25378816, 4096 00020000:00200000:2.0:1789760763.554313:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.554435:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6195, 6195] 00000008:00010000:2.0:1789760763.554464:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.554470:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760763.554482:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25374720, 25378816) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.554497:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25374720=0x1833000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.554558:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.554561:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25378816 00000020:00010000:2.0:1789760763.554570:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.554573:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.554576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.554580:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.554582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.554590:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.554599:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.554617:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760763.554754:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 25374720<>25378816 00000100:00100000:3.0:1789760763.554831:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495874816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760763.604414:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973512 gen 2 00000008:00010000:3.0:1789760763.604549:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760763.604559:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.604571:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.604685:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760763.604716:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495874816:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760763.605043:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6195, 6195] 00000080:00200000:2.0:1789760763.605061:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.605068:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.605084:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25378816 i_blocks 51209 00000020:00200000:2.0:1789760763.605088:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.605097:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.605132:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25378816, count: 4096 00000080:00200000:2.0:1789760763.605140:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25378816, bytes: 4096 00000080:00200000:2.0:1789760763.605154:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.605164:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.605166:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25378816, 25382912) 0 0 00000080:00200000:2.0:1789760763.605205:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.605223:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6196, 6196] 00020000:00200000:2.0:1789760763.605232:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25378816 chunk: [25378816, 25382912) 25382912, 4096 00020000:00200000:2.0:1789760763.605236:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.605257:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6196, 6196] 00000008:00010000:2.0:1789760763.605276:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.605282:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760763.605293:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25378816, 25382912) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.605308:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25378816=0x1834000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.606105:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.606111:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25382912 00000020:00010000:2.0:1789760763.606122:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.606126:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.606130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.606133:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.606136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.606146:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.606153:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.606725:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760763.607273:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 25378816<>25382912 00000100:00100000:2.0:1789760763.607354:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495874944:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760763.648343:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973513 gen 2 00000008:00010000:2.0:1789760763.648395:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760763.648404:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.648415:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.648457:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000080:00200000:3.0:1789760763.648478:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6196, 6196] 00000100:00100000:2.0:1789760763.648490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495874944:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.648499:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.648506:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.648522:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25382912 i_blocks 51209 00000020:00200000:3.0:1789760763.648526:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.648537:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.648573:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25382912, count: 4096 00000080:00200000:3.0:1789760763.648580:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25382912, bytes: 4096 00000080:00200000:3.0:1789760763.648591:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.648599:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.648602:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25382912, 25387008) 0 0 00000080:00200000:3.0:1789760763.648606:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.648623:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6197, 6197] 00020000:00200000:3.0:1789760763.648631:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25382912 chunk: [25382912, 25387008) 25387008, 4096 00020000:00200000:3.0:1789760763.648635:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.648654:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6197, 6197] 00000008:00010000:3.0:1789760763.648672:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.648678:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760763.648690:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25382912, 25387008) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.648705:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25382912=0x1835000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.648760:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.648828:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25387008 00000020:00010000:3.0:1789760763.648838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.648842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.648845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.648857:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.648860:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.648868:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.648876:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.648895:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760763.649029:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 25382912<>25387008 00000100:00100000:2.0:1789760763.649519:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760763.649540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495875072:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760763.707054:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973514 gen 2 00000008:00010000:2.0:1789760763.707105:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:2.0:1789760763.707114:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.707125:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.707264:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:2.0:1789760763.707294:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495875072:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760763.708831:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6197, 6197] 00000080:00200000:3.0:1789760763.708855:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.708862:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.708879:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25387008 i_blocks 51209 00000020:00200000:3.0:1789760763.708884:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.708897:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.708932:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25387008, count: 4096 00000080:00200000:3.0:1789760763.708941:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25387008, bytes: 4096 00000080:00200000:3.0:1789760763.708955:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.708964:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.708966:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25387008, 25391104) 0 0 00000080:00200000:3.0:1789760763.708971:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.708989:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6198, 6198] 00020000:00200000:3.0:1789760763.708998:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25387008 chunk: [25387008, 25391104) 25391104, 4096 00020000:00200000:3.0:1789760763.709002:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.709045:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6198, 6198] 00000008:00010000:3.0:1789760763.709065:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.709071:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760763.709082:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25387008, 25391104) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.709951:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25387008=0x1836000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.710398:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.710401:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25391104 00000020:00010000:3.0:1789760763.710412:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.710416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.710420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.710424:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.710427:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.710439:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.710459:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.710493:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760763.714655:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 25387008<>25391104 00000100:00100000:3.0:1789760763.714764:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760763.714792:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495875200:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760763.758734:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973515 gen 2 00000008:00010000:3.0:1789760763.758795:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760763.758804:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.758815:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.758838:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760763.758867:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495875200:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760763.758986:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6198, 6198] 00000080:00200000:2.0:1789760763.759013:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.759019:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.759036:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25391104 i_blocks 51209 00000020:00200000:2.0:1789760763.759039:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.759049:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.759083:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25391104, count: 4096 00000080:00200000:2.0:1789760763.759089:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25391104, bytes: 4096 00000080:00200000:2.0:1789760763.759098:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.759106:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.759109:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25391104, 25395200) 0 0 00000080:00200000:2.0:1789760763.759113:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.759128:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6199, 6199] 00020000:00200000:2.0:1789760763.759136:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25391104 chunk: [25391104, 25395200) 25395200, 4096 00020000:00200000:2.0:1789760763.759140:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.759161:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6199, 6199] 00000008:00010000:2.0:1789760763.759222:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.759228:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760763.759242:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25391104, 25395200) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.759258:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25391104=0x1837000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.759315:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.759319:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25395200 00000020:00010000:2.0:1789760763.759328:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.759331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.759334:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.759337:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.759340:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.759348:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.759355:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.759370:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760763.759509:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 25391104<>25395200 00000100:00100000:2.0:1789760763.759553:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760763.759580:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495875328:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760763.789269:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495875456:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760763.789396:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495875712:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760763.789437:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.789454:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495875584:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760763.794880:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495875584:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760763.794901:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495875456:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760763.794914:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495875712:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760763.826428:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973516 gen 2 00000008:00010000:2.0:1789760763.826493:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760763.826502:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.826513:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.826538:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760763.826578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495875328:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760763.826750:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6199, 6199] 00000080:00200000:0.0:1789760763.826766:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.826772:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.826788:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25395200 i_blocks 51209 00000020:00200000:0.0:1789760763.826791:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.826802:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.826833:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25395200, count: 4096 00000080:00200000:0.0:1789760763.826841:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25395200, bytes: 4096 00000080:00200000:0.0:1789760763.826854:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.826863:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.826865:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25395200, 25399296) 0 0 00000080:00200000:0.0:1789760763.826869:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.826886:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6200, 6200] 00020000:00200000:0.0:1789760763.826894:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25395200 chunk: [25395200, 25399296) 25399296, 4096 00020000:00200000:0.0:1789760763.826900:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.826934:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6200, 6200] 00000008:00010000:0.0:1789760763.826955:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.826961:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760763.826974:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25395200, 25399296) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.826989:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25395200=0x1838000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.827067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.827071:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25399296 00000020:00010000:0.0:1789760763.827080:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.827083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.827087:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.827090:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.827093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.827101:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.827110:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.827127:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760763.827727:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 25395200<>25399296 00000100:00100000:1.0:1789760763.827801:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495875840:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760763.863312:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973517 gen 2 00000008:00010000:1.0:1789760763.863494:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760763.863504:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.863516:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.863541:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760763.863581:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495875840:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760763.863705:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6200, 6200] 00000080:00200000:2.0:1789760763.863722:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.863727:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.863743:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25399296 i_blocks 51209 00000020:00200000:2.0:1789760763.863746:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.863756:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.863790:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25399296, count: 4096 00000080:00200000:2.0:1789760763.863797:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25399296, bytes: 4096 00000080:00200000:2.0:1789760763.863808:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.863816:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.863819:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25399296, 25403392) 0 0 00000080:00200000:2.0:1789760763.863823:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.863848:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6201, 6201] 00020000:00200000:2.0:1789760763.863857:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25399296 chunk: [25399296, 25403392) 25403392, 4096 00020000:00200000:2.0:1789760763.863863:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.863884:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6201, 6201] 00000008:00010000:2.0:1789760763.863903:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.863908:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760763.863920:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25399296, 25403392) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.863934:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25399296=0x1839000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.864001:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.864005:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25403392 00000020:00010000:2.0:1789760763.864014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.864025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.864028:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.864032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.864035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.864042:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.864049:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.864066:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760763.864235:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 25399296<>25403392 00000100:00100000:3.0:1789760763.864321:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495875968:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760763.903249:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973518 gen 2 00000008:00010000:2.0:1789760763.903385:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760763.903394:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.903559:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.903687:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760763.903717:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495875968:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760763.903835:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6201, 6201] 00000080:00200000:1.0:1789760763.903851:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760763.903858:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760763.903874:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25403392 i_blocks 51209 00000020:00200000:1.0:1789760763.903878:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760763.903887:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760763.903921:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25403392, count: 4096 00000080:00200000:1.0:1789760763.903927:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25403392, bytes: 4096 00000080:00200000:1.0:1789760763.903937:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760763.903946:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760763.903949:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25403392, 25407488) 0 0 00000080:00200000:1.0:1789760763.903953:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760763.903971:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6202, 6202] 00020000:00200000:1.0:1789760763.903978:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25403392 chunk: [25403392, 25407488) 25407488, 4096 00020000:00200000:1.0:1789760763.903983:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760763.904002:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6202, 6202] 00000008:00010000:1.0:1789760763.904020:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760763.904026:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760763.904037:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25403392, 25407488) append 0 append lockpos 0 00000080:00200000:1.0:1789760763.904050:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25403392=0x183a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760763.904101:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760763.904103:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25407488 00000020:00010000:1.0:1789760763.904111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760763.904115:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.904117:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.904121:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760763.904124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760763.904131:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760763.904138:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.904154:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760763.904297:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25403392<>25407488 00000100:00100000:1.0:1789760763.904341:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.904358:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495876096:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760763.937238:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973519 gen 2 00000008:00010000:1.0:1789760763.937288:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760763.937295:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.937306:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.937328:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760763.937366:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495876096:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760763.937592:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6202, 6202] 00000080:00200000:0.0:1789760763.937609:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.937615:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.937632:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25407488 i_blocks 51209 00000020:00200000:0.0:1789760763.937636:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.937645:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.937680:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25407488, count: 4096 00000080:00200000:0.0:1789760763.937688:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25407488, bytes: 4096 00000080:00200000:0.0:1789760763.937699:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.937708:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.937710:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25407488, 25411584) 0 0 00000080:00200000:0.0:1789760763.937714:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.937731:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6203, 6203] 00020000:00200000:0.0:1789760763.937739:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25407488 chunk: [25407488, 25411584) 25411584, 4096 00020000:00200000:0.0:1789760763.937744:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.937774:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6203, 6203] 00000008:00010000:0.0:1789760763.937792:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.937798:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760763.937809:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25407488, 25411584) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.937823:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25407488=0x183b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.937894:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.937897:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25411584 00000020:00010000:0.0:1789760763.937907:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.937911:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.937914:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.937918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.937921:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.937928:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.937936:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.937954:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760763.938081:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 25407488<>25411584 00000100:00100000:0.0:1789760763.938127:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760763.938143:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876224:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760763.977112:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973520 gen 2 00000008:00010000:1.0:1789760763.977162:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760763.977292:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.977304:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.977363:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760763.977392:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876224:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760763.977530:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6203, 6203] 00000080:00200000:2.0:1789760763.977548:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.977554:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.977570:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25411584 i_blocks 51209 00000020:00200000:2.0:1789760763.977574:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.977583:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.977615:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25411584, count: 4096 00000080:00200000:2.0:1789760763.977623:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25411584, bytes: 4096 00000080:00200000:2.0:1789760763.977633:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.977643:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.977645:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25411584, 25415680) 0 0 00000080:00200000:2.0:1789760763.977650:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.977666:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6204, 6204] 00020000:00200000:2.0:1789760763.977675:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25411584 chunk: [25411584, 25415680) 25415680, 4096 00020000:00200000:2.0:1789760763.977680:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.977701:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6204, 6204] 00000008:00010000:2.0:1789760763.977718:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.977725:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760763.977736:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25411584, 25415680) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.977750:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25411584=0x183c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.977925:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.977928:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25415680 00000020:00010000:2.0:1789760763.977938:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.977942:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.977945:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.977948:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.977950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.977958:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.977964:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.977980:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760763.978101:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 25411584<>25415680 00000100:00100000:3.0:1789760763.978191:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495876352:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760764.008907:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973521 gen 2 00000008:00010000:3.0:1789760764.008957:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760764.008966:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.008987:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.009007:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760764.009037:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495876352:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760764.009204:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6204, 6204] 00000080:00200000:0.0:1789760764.009221:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.009228:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.009244:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25415680 i_blocks 51209 00000020:00200000:0.0:1789760764.009248:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.009257:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.009292:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25415680, count: 4096 00000080:00200000:0.0:1789760764.009299:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25415680, bytes: 4096 00000080:00200000:0.0:1789760764.009310:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.009319:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.009321:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25415680, 25419776) 0 0 00000080:00200000:0.0:1789760764.009325:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.009343:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6205, 6205] 00020000:00200000:0.0:1789760764.009352:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25415680 chunk: [25415680, 25419776) 25419776, 4096 00020000:00200000:0.0:1789760764.009356:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.009375:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6205, 6205] 00000008:00010000:0.0:1789760764.009392:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.009397:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760764.009407:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25415680, 25419776) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.009424:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25415680=0x183d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.009499:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.009503:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25419776 00000020:00010000:0.0:1789760764.009511:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.009515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.009518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.009522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.009526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.009534:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.009541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.009557:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760764.009688:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 25415680<>25419776 00000100:00100000:1.0:1789760764.010001:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876480:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760764.062815:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973522 gen 2 00000008:00010000:1.0:1789760764.062983:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760764.062992:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.063003:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.063024:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760764.063059:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876480:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760764.063240:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6205, 6205] 00000080:00200000:0.0:1789760764.063259:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.063265:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.063283:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25419776 i_blocks 51209 00000020:00200000:0.0:1789760764.063287:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.063296:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.063330:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25419776, count: 4096 00000080:00200000:0.0:1789760764.063337:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25419776, bytes: 4096 00000080:00200000:0.0:1789760764.063348:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.063357:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.063359:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25419776, 25423872) 0 0 00000080:00200000:0.0:1789760764.063363:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.063379:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6206, 6206] 00020000:00200000:0.0:1789760764.063387:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25419776 chunk: [25419776, 25423872) 25423872, 4096 00020000:00200000:0.0:1789760764.063391:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.063409:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6206, 6206] 00000008:00010000:0.0:1789760764.063494:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.063501:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760764.063513:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25419776, 25423872) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.063528:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25419776=0x183e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.063585:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.063588:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25423872 00000020:00010000:0.0:1789760764.063598:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.063601:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.063605:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.063609:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.063613:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.063621:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.063628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.063644:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760764.064062:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 25419776<>25423872 00000100:00100000:1.0:1789760764.064261:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760764.064285:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876608:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760764.109605:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760764.109622:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.109633:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.109657:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760764.109689:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876608:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760764.109694:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6206, 6206] 00000080:00200000:0.0:1789760764.109902:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.109910:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.109927:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25423872 i_blocks 51209 00000020:00200000:0.0:1789760764.109931:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.109942:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.109980:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25423872, count: 4096 00000080:00200000:0.0:1789760764.109987:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25423872, bytes: 4096 00000080:00200000:0.0:1789760764.109996:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.110006:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.110009:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25423872, 25427968) 0 0 00000080:00200000:0.0:1789760764.110013:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.110030:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6207, 6207] 00020000:00200000:0.0:1789760764.110039:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25423872 chunk: [25423872, 25427968) 25427968, 4096 00020000:00200000:0.0:1789760764.110055:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.110077:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6207, 6207] 00000008:00010000:0.0:1789760764.110095:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.110102:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760764.110113:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25423872, 25427968) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.110129:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25423872=0x183f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.110227:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.110231:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25427968 00000020:00010000:0.0:1789760764.110240:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.110244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.110247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.110250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.110254:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.110261:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.110269:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.110286:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760764.110405:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 25423872<>25427968 00000100:00100000:1.0:1789760764.110667:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876736:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760764.151257:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973524 gen 2 00000008:00010000:1.0:1789760764.151319:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760764.151327:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.151337:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.151361:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760764.151381:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876736:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760764.151842:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6207, 6207] 00000080:00200000:0.0:1789760764.151860:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.151865:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.151880:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25427968 i_blocks 51209 00000020:00200000:0.0:1789760764.151883:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.151893:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.151926:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25427968, count: 4096 00000080:00200000:0.0:1789760764.151933:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25427968, bytes: 4096 00000080:00200000:0.0:1789760764.151946:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.151954:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.151957:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25427968, 25432064) 0 0 00000080:00200000:0.0:1789760764.151961:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.151977:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6208, 6208] 00020000:00200000:0.0:1789760764.151984:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25427968 chunk: [25427968, 25432064) 25432064, 4096 00020000:00200000:0.0:1789760764.151989:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.152008:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6208, 6208] 00000008:00010000:0.0:1789760764.152025:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.152031:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760764.152043:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25427968, 25432064) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.152058:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25427968=0x1840000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.152118:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.152121:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25432064 00000020:00010000:0.0:1789760764.152130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.152134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.152136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.152140:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.152144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.152151:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.152204:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.152220:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760764.152348:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 25427968<>25432064 00000100:00100000:1.0:1789760764.152652:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760764.152752:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876864:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760764.190409:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973525 gen 2 00000008:00010000:1.0:1789760764.190470:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760764.190479:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.190705:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.190828:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760764.190860:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876864:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760764.191010:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6208, 6208] 00000080:00200000:0.0:1789760764.191027:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.191033:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.191049:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25432064 i_blocks 51209 00000020:00200000:0.0:1789760764.191054:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.191063:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.191204:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25432064, count: 4096 00000080:00200000:0.0:1789760764.191213:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25432064, bytes: 4096 00000080:00200000:0.0:1789760764.191228:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.191237:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.191239:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25432064, 25436160) 0 0 00000080:00200000:0.0:1789760764.191244:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.191260:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6209, 6209] 00020000:00200000:0.0:1789760764.191269:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25432064 chunk: [25432064, 25436160) 25436160, 4096 00020000:00200000:0.0:1789760764.191273:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.191293:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6209, 6209] 00000008:00010000:0.0:1789760764.191309:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.191316:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760764.191328:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25432064, 25436160) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.191342:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25432064=0x1841000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.191410:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.191413:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25436160 00000020:00010000:0.0:1789760764.191421:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.191425:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.191428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.191431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.191434:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.191451:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.191458:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.191475:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760764.192455:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 25432064<>25436160 00000100:00100000:1.0:1789760764.192893:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876992:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760764.250758:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760764.250775:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.250786:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.250846:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760764.250880:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495876992:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760764.254004:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6209, 6209] 00000080:00200000:0.0:1789760764.254034:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.254043:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.254062:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25436160 i_blocks 51209 00000020:00200000:0.0:1789760764.254067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.254082:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.254116:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25436160, count: 4096 00000080:00200000:0.0:1789760764.254125:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25436160, bytes: 4096 00000080:00200000:0.0:1789760764.254139:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.254149:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.254151:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25436160, 25440256) 0 0 00000080:00200000:0.0:1789760764.254156:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.254190:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6210, 6210] 00020000:00200000:0.0:1789760764.254199:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25436160 chunk: [25436160, 25440256) 25440256, 4096 00020000:00200000:0.0:1789760764.254204:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.254804:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6210, 6210] 00000008:00010000:0.0:1789760764.254825:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.254841:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760764.254853:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25436160, 25440256) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.254868:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25436160=0x1842000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.254935:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.254937:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25440256 00000020:00010000:0.0:1789760764.254946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.254950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.254953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.254956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.254960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.254969:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.254976:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.254995:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760764.255124:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25436160<>25440256 00000100:00100000:1.0:1789760764.255645:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760764.255673:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495877120:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760764.295096:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973527 gen 2 00000008:00010000:1.0:1789760764.295163:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760764.295201:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.295213:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.295292:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760764.295323:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495877120:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760764.295400:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6210, 6210] 00000080:00200000:2.0:1789760764.295624:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.295633:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.295650:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25440256 i_blocks 51209 00000020:00200000:2.0:1789760764.295654:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.295663:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.295698:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25440256, count: 4096 00000080:00200000:2.0:1789760764.295706:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25440256, bytes: 4096 00000080:00200000:2.0:1789760764.295717:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.295726:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.295729:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25440256, 25444352) 0 0 00000080:00200000:2.0:1789760764.295734:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.295750:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6211, 6211] 00020000:00200000:2.0:1789760764.295923:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25440256 chunk: [25440256, 25444352) 25444352, 4096 00020000:00200000:2.0:1789760764.295928:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.295949:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6211, 6211] 00000008:00010000:2.0:1789760764.295965:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.295970:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760764.295981:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25440256, 25444352) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.295994:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25440256=0x1843000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.296048:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.296050:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25444352 00000020:00010000:2.0:1789760764.296058:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.296061:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.296064:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.296068:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.296071:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.296077:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.296084:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.296103:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760764.296681:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 25440256<>25444352 00000100:00100000:3.0:1789760764.296746:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760764.297117:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495877248:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760764.341041:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760764.341056:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.341068:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.341095:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760764.341127:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495877248:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760764.341720:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6211, 6211] 00000080:00200000:1.0:1789760764.341739:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760764.341745:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760764.341760:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25444352 i_blocks 51209 00000020:00200000:1.0:1789760764.341764:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760764.341775:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760764.341811:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25444352, count: 4096 00000080:00200000:1.0:1789760764.341819:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25444352, bytes: 4096 00000080:00200000:1.0:1789760764.341831:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760764.341840:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760764.341843:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25444352, 25448448) 0 0 00000080:00200000:1.0:1789760764.341847:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760764.341864:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6212, 6212] 00020000:00200000:1.0:1789760764.341872:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25444352 chunk: [25444352, 25448448) 25448448, 4096 00020000:00200000:1.0:1789760764.341877:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760764.341899:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6212, 6212] 00000008:00010000:1.0:1789760764.341916:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760764.341922:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760764.341934:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25444352, 25448448) append 0 append lockpos 0 00000080:00200000:1.0:1789760764.341949:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25444352=0x1844000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760764.342019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760764.342024:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25448448 00000020:00010000:1.0:1789760764.342034:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760764.342038:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.342041:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.342044:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760764.342047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760764.342055:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760764.342061:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760764.342079:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760764.342257:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25444352<>25448448 00000100:00100000:1.0:1789760764.342299:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760764.342319:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495877376:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760764.363647:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760764.398267:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973529 gen 2 00000008:00010000:1.0:1789760764.398328:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760764.398337:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.398348:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.398373:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760764.398410:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495877376:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760764.398664:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6212, 6212] 00000080:00200000:3.0:1789760764.398681:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.398688:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.398704:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25448448 i_blocks 51209 00000020:00200000:3.0:1789760764.398708:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.398717:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.398859:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25448448, count: 4096 00000080:00200000:3.0:1789760764.398868:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25448448, bytes: 4096 00000080:00200000:3.0:1789760764.398881:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.398891:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.398893:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25448448, 25452544) 0 0 00000080:00200000:3.0:1789760764.398897:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.398913:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6213, 6213] 00020000:00200000:3.0:1789760764.398922:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25448448 chunk: [25448448, 25452544) 25452544, 4096 00020000:00200000:3.0:1789760764.398926:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.398953:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6213, 6213] 00000008:00010000:3.0:1789760764.398970:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.398976:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760764.398986:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25448448, 25452544) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.398999:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25448448=0x1845000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.399052:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.399055:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25452544 00000020:00010000:3.0:1789760764.399062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.399066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.399069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.399073:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.399076:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.399082:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.399088:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.399104:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760764.399258:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 25448448<>25452544 00000100:00100000:2.0:1789760764.399608:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760764.399629:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495877504:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760764.442873:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973530 gen 2 00000008:00010000:2.0:1789760764.442926:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760764.442934:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.442944:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.442965:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760764.442996:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495877504:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760764.443109:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6213, 6213] 00000080:00200000:3.0:1789760764.443126:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.443132:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.443148:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25452544 i_blocks 51209 00000020:00200000:3.0:1789760764.443151:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.443161:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.443216:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25452544, count: 4096 00000080:00200000:3.0:1789760764.443224:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25452544, bytes: 4096 00000080:00200000:3.0:1789760764.443235:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.443244:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.443247:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25452544, 25456640) 0 0 00000080:00200000:3.0:1789760764.443251:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.443268:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6214, 6214] 00020000:00200000:3.0:1789760764.443276:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25452544 chunk: [25452544, 25456640) 25456640, 4096 00020000:00200000:3.0:1789760764.443281:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.443302:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6214, 6214] 00000008:00010000:3.0:1789760764.443319:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.443324:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760764.443335:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25452544, 25456640) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.443349:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25452544=0x1846000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.443404:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.443407:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25456640 00000020:00010000:3.0:1789760764.443545:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.443550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.443553:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.443557:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.443560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.443568:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.443575:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.443592:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760764.443959:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 25452544<>25456640 00000100:00100000:2.0:1789760764.444087:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495877632:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760764.511666:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760764.511740:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.511752:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.511833:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:2.0:1789760764.511866:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495877632:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760764.512031:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6214, 6214] 00000080:00200000:3.0:1789760764.512048:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.512055:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.512071:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25456640 i_blocks 51209 00000020:00200000:3.0:1789760764.512075:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.512086:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.512119:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25456640, count: 4096 00000080:00200000:3.0:1789760764.512126:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25456640, bytes: 4096 00000080:00200000:3.0:1789760764.512138:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.512147:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.512150:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25456640, 25460736) 0 0 00000080:00200000:3.0:1789760764.512154:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.512555:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6215, 6215] 00020000:00200000:3.0:1789760764.512565:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25456640 chunk: [25456640, 25460736) 25460736, 4096 00020000:00200000:3.0:1789760764.512570:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.512670:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6215, 6215] 00000008:00010000:2.0:1789760764.512843:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.512850:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760764.512865:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25456640, 25460736) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.512880:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25456640=0x1847000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.513574:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.513579:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25460736 00000020:00010000:2.0:1789760764.513590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.513594:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.513597:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.513601:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.513603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.513615:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.513624:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.513647:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760764.513905:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 25456640<>25460736 00000100:00100000:2.0:1789760764.514040:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760764.514064:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495877760:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760764.594630:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973532 gen 2 00000008:00010000:2.0:1789760764.594691:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760764.594700:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.594711:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.594739:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760764.594775:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495877760:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760764.595293:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6215, 6215] 00000080:00200000:1.0:1789760764.595310:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760764.595316:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760764.595332:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25460736 i_blocks 51209 00000020:00200000:1.0:1789760764.595336:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760764.595345:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760764.595375:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25460736, count: 4096 00000080:00200000:1.0:1789760764.595382:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25460736, bytes: 4096 00000080:00200000:1.0:1789760764.595396:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760764.595406:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760764.595409:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25460736, 25464832) 0 0 00000080:00200000:1.0:1789760764.595414:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760764.595432:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6216, 6216] 00020000:00200000:1.0:1789760764.595450:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25460736 chunk: [25460736, 25464832) 25464832, 4096 00020000:00200000:1.0:1789760764.595455:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760764.595474:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6216, 6216] 00000008:00010000:1.0:1789760764.595503:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760764.595508:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760764.595521:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25460736, 25464832) append 0 append lockpos 0 00000080:00200000:1.0:1789760764.595536:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25460736=0x1848000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760764.595594:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760764.595597:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25464832 00000020:00010000:1.0:1789760764.595605:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760764.595608:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.595612:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.595615:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760764.595618:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760764.595625:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760764.595632:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760764.595650:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760764.595779:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25460736<>25464832 00000100:00100000:0.0:1789760764.596061:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760764.596082:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495877888:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760764.650516:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973533 gen 2 00000008:00010000:1.0:1789760764.650569:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760764.650579:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.650591:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.650612:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760764.650643:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495877888:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760764.650659:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6216, 6216] 00000080:00200000:2.0:1789760764.650680:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.650687:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.650715:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25464832 i_blocks 51209 00000020:00200000:2.0:1789760764.650720:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.650734:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.650768:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25464832, count: 4096 00000080:00200000:2.0:1789760764.650775:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25464832, bytes: 4096 00000080:00200000:2.0:1789760764.650785:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.650794:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.650797:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25464832, 25468928) 0 0 00000080:00200000:2.0:1789760764.650801:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.650818:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6217, 6217] 00020000:00200000:2.0:1789760764.650827:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25464832 chunk: [25464832, 25468928) 25468928, 4096 00020000:00200000:2.0:1789760764.650832:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.650850:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6217, 6217] 00000008:00010000:2.0:1789760764.650866:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.650872:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760764.650881:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25464832, 25468928) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.650895:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25464832=0x1849000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.650948:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.650951:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25468928 00000020:00010000:2.0:1789760764.650959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.650962:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.650965:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.650969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.650972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.650980:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.650985:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.650999:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760764.651120:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 25464832<>25468928 00000100:00100000:3.0:1789760764.651943:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495878016:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760764.687492:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973534 gen 2 00000008:00010000:3.0:1789760764.687542:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760764.687551:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.687562:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.687590:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760764.687622:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495878016:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760764.687696:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6217, 6217] 00000080:00200000:2.0:1789760764.687713:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.687720:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.687735:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25468928 i_blocks 51209 00000020:00200000:2.0:1789760764.687739:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.687748:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.687783:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25468928, count: 4096 00000080:00200000:2.0:1789760764.687790:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25468928, bytes: 4096 00000080:00200000:2.0:1789760764.687799:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.687807:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.687810:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25468928, 25473024) 0 0 00000080:00200000:2.0:1789760764.687814:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.687830:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6218, 6218] 00020000:00200000:2.0:1789760764.687839:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25468928 chunk: [25468928, 25473024) 25473024, 4096 00020000:00200000:2.0:1789760764.687844:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.687863:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6218, 6218] 00000008:00010000:2.0:1789760764.687881:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.687887:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760764.687901:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25468928, 25473024) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.687915:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25468928=0x184a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.687970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.687973:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25473024 00000020:00010000:2.0:1789760764.687982:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.687986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.687989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.687993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.687996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.688004:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.688010:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.688026:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760764.688247:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 25468928<>25473024 00000100:00100000:3.0:1789760764.689251:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760764.689276:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495878144:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760764.735251:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973535 gen 2 00000008:00010000:3.0:1789760764.735298:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760764.735306:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.735316:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.735336:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760764.735362:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495878144:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760764.736330:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6218, 6218] 00000080:00200000:2.0:1789760764.736350:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.736357:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.736374:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25473024 i_blocks 51209 00000020:00200000:2.0:1789760764.736379:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.736390:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.736422:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25473024, count: 4096 00000080:00200000:2.0:1789760764.736430:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25473024, bytes: 4096 00000080:00200000:2.0:1789760764.736458:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.736467:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.736469:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25473024, 25477120) 0 0 00000080:00200000:2.0:1789760764.736474:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.736491:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6219, 6219] 00020000:00200000:2.0:1789760764.736500:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25473024 chunk: [25473024, 25477120) 25477120, 4096 00020000:00200000:2.0:1789760764.736504:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.736525:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6219, 6219] 00000008:00010000:2.0:1789760764.736543:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.736549:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760764.736560:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25473024, 25477120) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.736575:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25473024=0x184b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.736638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.736641:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25477120 00000020:00010000:2.0:1789760764.736649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.736653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.736656:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.736659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.736662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.736671:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.736678:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.736696:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760764.736821:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 25473024<>25477120 00000100:00100000:3.0:1789760764.736902:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495878272:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760764.782790:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760764.782802:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.782812:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.782830:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760764.782856:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495878272:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760764.782906:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6219, 6219] 00000080:00200000:2.0:1789760764.782930:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.782938:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.782954:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25477120 i_blocks 51209 00000020:00200000:2.0:1789760764.782959:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.782972:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.783005:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25477120, count: 4096 00000080:00200000:2.0:1789760764.783013:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25477120, bytes: 4096 00000080:00200000:2.0:1789760764.783024:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.783033:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.783035:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25477120, 25481216) 0 0 00000080:00200000:2.0:1789760764.783039:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.783069:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6220, 6220] 00020000:00200000:2.0:1789760764.783077:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25477120 chunk: [25477120, 25481216) 25481216, 4096 00020000:00200000:2.0:1789760764.783081:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.783102:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6220, 6220] 00000008:00010000:2.0:1789760764.783122:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.783128:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760764.783139:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25477120, 25481216) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.783153:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25477120=0x184c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.783455:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.783459:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25481216 00000020:00010000:2.0:1789760764.783469:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.783473:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.783476:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.783479:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.783482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.783491:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.785406:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.785463:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760764.787355:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 25477120<>25481216 00000100:00100000:2.0:1789760764.787463:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495878400:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760764.825753:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973537 gen 2 00000008:00010000:2.0:1789760764.825816:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760764.825823:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.825834:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.825858:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760764.825886:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495878400:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760764.825989:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6220, 6220] 00000080:00200000:3.0:1789760764.826013:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.826019:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.826034:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25481216 i_blocks 51209 00000020:00200000:3.0:1789760764.826037:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.826047:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.826080:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25481216, count: 4096 00000080:00200000:3.0:1789760764.826095:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25481216, bytes: 4096 00000080:00200000:3.0:1789760764.826107:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.826115:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.826118:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25481216, 25485312) 0 0 00000080:00200000:3.0:1789760764.826122:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.826138:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6221, 6221] 00020000:00200000:3.0:1789760764.826146:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25481216 chunk: [25481216, 25485312) 25485312, 4096 00020000:00200000:3.0:1789760764.826151:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.826208:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6221, 6221] 00000008:00010000:3.0:1789760764.826224:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.826229:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760764.826241:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25481216, 25485312) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.826253:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25481216=0x184d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.826579:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.826582:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25485312 00000020:00010000:3.0:1789760764.826591:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.826595:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.826599:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.826611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.826614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.826623:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.826630:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.826646:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760764.826766:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 25481216<>25485312 00000100:00100000:3.0:1789760764.826809:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760764.826825:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495878528:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760764.881087:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760764.881101:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.881112:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.881139:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760764.881166:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495878528:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760764.881333:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6221, 6221] 00000080:00200000:1.0:1789760764.881350:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760764.881357:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760764.881372:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25485312 i_blocks 51209 00000020:00200000:1.0:1789760764.881376:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760764.881385:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760764.881687:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25485312, count: 4096 00000080:00200000:1.0:1789760764.881696:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25485312, bytes: 4096 00000080:00200000:1.0:1789760764.881706:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760764.881715:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760764.881718:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25485312, 25489408) 0 0 00000080:00200000:1.0:1789760764.881722:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760764.881739:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6222, 6222] 00020000:00200000:1.0:1789760764.881747:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25485312 chunk: [25485312, 25489408) 25489408, 4096 00020000:00200000:1.0:1789760764.881751:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760764.882133:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6222, 6222] 00000008:00010000:1.0:1789760764.882151:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760764.882157:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760764.882189:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25485312, 25489408) append 0 append lockpos 0 00000080:00200000:1.0:1789760764.882205:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25485312=0x184e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760764.882258:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760764.882261:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25489408 00000020:00010000:1.0:1789760764.882269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760764.882272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.882275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.882279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760764.882281:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760764.882289:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760764.882295:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760764.882310:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760764.882505:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25485312<>25489408 00000100:00100000:0.0:1789760764.882676:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495878656:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760764.920015:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973539 gen 2 00000008:00010000:1.0:1789760764.920122:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760764.920131:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.920140:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.920281:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760764.920307:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495878656:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760764.921909:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6222, 6222] 00000080:00200000:3.0:1789760764.921932:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.921938:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.922138:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25489408 i_blocks 51209 00000020:00200000:2.0:1789760764.922145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.922159:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.922212:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25489408, count: 4096 00000080:00200000:2.0:1789760764.922222:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25489408, bytes: 4096 00000080:00200000:2.0:1789760764.922235:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.922244:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.922246:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25489408, 25493504) 0 0 00000080:00200000:2.0:1789760764.922251:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.922268:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6223, 6223] 00020000:00200000:2.0:1789760764.922276:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25489408 chunk: [25489408, 25493504) 25493504, 4096 00020000:00200000:2.0:1789760764.922281:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.922302:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6223, 6223] 00000008:00010000:2.0:1789760764.922322:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.922327:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760764.922339:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25489408, 25493504) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.922353:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25489408=0x184f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.922416:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.922420:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25493504 00000020:00010000:2.0:1789760764.922428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.922802:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.922806:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.922810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.922813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.922822:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.922830:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.922890:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760764.923125:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 25489408<>25493504 00000100:00100000:2.0:1789760764.923221:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760764.923242:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495878784:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760764.958248:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973540 gen 2 00000008:00010000:2.0:1789760764.958294:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760764.958302:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.958313:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.958340:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760764.958370:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495878784:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760764.958548:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6223, 6223] 00000080:00200000:1.0:1789760764.958565:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760764.958570:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760764.958587:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25493504 i_blocks 51209 00000020:00200000:1.0:1789760764.958590:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760764.958662:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760764.958695:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25493504, count: 4096 00000080:00200000:1.0:1789760764.958703:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25493504, bytes: 4096 00000080:00200000:1.0:1789760764.958713:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760764.958721:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760764.958724:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25493504, 25497600) 0 0 00000080:00200000:1.0:1789760764.958728:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760764.958744:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6224, 6224] 00020000:00200000:1.0:1789760764.958807:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25493504 chunk: [25493504, 25497600) 25497600, 4096 00020000:00200000:1.0:1789760764.958812:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760764.958939:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6224, 6224] 00000008:00010000:1.0:1789760764.958955:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760764.958961:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760764.958973:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25493504, 25497600) append 0 append lockpos 0 00000080:00200000:1.0:1789760764.958986:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25493504=0x1850000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760764.959037:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760764.959040:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25497600 00000020:00010000:1.0:1789760764.959048:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760764.959051:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.959054:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.959058:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760764.959062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760764.959068:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760764.959074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760764.959088:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760764.959300:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25493504<>25497600 00000100:00100000:0.0:1789760764.959349:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760764.959964:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495878912:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760764.996208:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760764.996222:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.996233:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.996288:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760764.996314:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495878912:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760764.996655:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6224, 6224] 00000080:00200000:0.0:1789760764.996673:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.996678:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.996694:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25497600 i_blocks 51209 00000020:00200000:0.0:1789760764.996698:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.996707:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.996740:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25497600, count: 4096 00000080:00200000:0.0:1789760764.996747:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25497600, bytes: 4096 00000080:00200000:0.0:1789760764.996807:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.996816:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.996818:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25497600, 25501696) 0 0 00000080:00200000:0.0:1789760764.996877:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.996896:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6225, 6225] 00020000:00200000:0.0:1789760764.996904:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25497600 chunk: [25497600, 25501696) 25501696, 4096 00020000:00200000:0.0:1789760764.996909:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.996928:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6225, 6225] 00000008:00010000:0.0:1789760764.996943:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.996949:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760764.996959:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25497600, 25501696) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.996972:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25497600=0x1851000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.997024:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.997027:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25501696 00000020:00010000:0.0:1789760764.997034:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.997038:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.997041:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.997044:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.997047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.997053:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.997060:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.997076:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760764.997282:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 25497600<>25501696 00000100:00100000:1.0:1789760764.997339:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495879040:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.037587:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973541 gen 2 00000100:00100000:1.0:1789760765.038365:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211834e00 x1876699495879040/t4294973542(4294973542) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760780 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760765.038418:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760765.038426:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.038526:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.038597:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760765.038799:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495879040:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.039009:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6225, 6225] 00000080:00200000:0.0:1789760765.039028:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.039035:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.039050:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25501696 i_blocks 51209 00000020:00200000:0.0:1789760765.039054:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.039065:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.039098:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25501696, count: 4096 00000080:00200000:0.0:1789760765.039105:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25501696, bytes: 4096 00000080:00200000:0.0:1789760765.039120:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.039129:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.039131:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25501696, 25505792) 0 0 00000080:00200000:0.0:1789760765.039135:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.039152:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6226, 6226] 00020000:00200000:0.0:1789760765.039160:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25501696 chunk: [25501696, 25505792) 25505792, 4096 00020000:00200000:0.0:1789760765.039165:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.039459:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6226, 6226] 00000008:00010000:0.0:1789760765.039478:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.039484:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.039497:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25501696, 25505792) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.039513:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25501696=0x1852000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.039569:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.039572:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25505792 00000020:00010000:0.0:1789760765.039581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.039585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.039588:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.039591:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.039594:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.039802:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.039810:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.040167:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760765.040676:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 25501696<>25505792 00000100:00100000:1.0:1789760765.040732:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760765.040815:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495879168:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.065681:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973543 gen 2 00000008:00010000:1.0:1789760765.065751:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760765.065759:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.065769:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.065856:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760765.065887:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495879168:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760765.066124:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6226, 6226] 00000080:00200000:2.0:1789760765.066143:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760765.066149:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760765.066166:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25505792 i_blocks 51209 00000020:00200000:2.0:1789760765.066233:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760765.066244:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760765.066277:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25505792, count: 4096 00000080:00200000:2.0:1789760765.066284:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25505792, bytes: 4096 00000080:00200000:2.0:1789760765.066296:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760765.066304:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760765.066307:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25505792, 25509888) 0 0 00000080:00200000:2.0:1789760765.066312:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760765.066329:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6227, 6227] 00020000:00200000:2.0:1789760765.066338:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25505792 chunk: [25505792, 25509888) 25509888, 4096 00020000:00200000:2.0:1789760765.066341:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760765.066358:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6227, 6227] 00000008:00010000:2.0:1789760765.066372:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760765.066378:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760765.066389:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25505792, 25509888) append 0 append lockpos 0 00000080:00200000:2.0:1789760765.066482:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25505792=0x1853000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760765.066545:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760765.066548:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25509888 00000020:00010000:2.0:1789760765.066557:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760765.066561:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.066563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.066567:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760765.066570:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760765.066577:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760765.066584:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760765.066688:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760765.067317:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 25505792<>25509888 00000100:00100000:2.0:1789760765.067362:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760765.067380:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495879296:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760765.111981:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973544 gen 2 00000008:00010000:3.0:1789760765.112029:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760765.112038:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760765.112049:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760765.112073:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760765.112102:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495879296:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.112266:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6227, 6227] 00000080:00200000:0.0:1789760765.112282:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.112287:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.112302:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25509888 i_blocks 51209 00000020:00200000:0.0:1789760765.112305:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.112314:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.112347:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25509888, count: 4096 00000080:00200000:0.0:1789760765.112354:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25509888, bytes: 4096 00000080:00200000:0.0:1789760765.112364:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.112373:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.112375:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25509888, 25513984) 0 0 00000080:00200000:0.0:1789760765.112380:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.112397:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6228, 6228] 00020000:00200000:0.0:1789760765.112405:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25509888 chunk: [25509888, 25513984) 25513984, 4096 00020000:00200000:0.0:1789760765.112410:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.112438:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6228, 6228] 00000008:00010000:0.0:1789760765.112457:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.112463:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.112474:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25509888, 25513984) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.112489:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25509888=0x1854000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.112545:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.112548:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25513984 00000020:00010000:0.0:1789760765.112556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.112559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.112562:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.112566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2065ce5d8 (ffff8dd2183767e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.112569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd2183767e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.112577:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.112583:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.112598:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760765.112719:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 25509888<>25513984 00000100:00100000:0.0:1789760765.112767:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760765.112782:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495879424:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.161366:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973545 gen 2 00000008:00010000:1.0:1789760765.161410:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760765.161418:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.161429:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.161512:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760765.161541:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495879424:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760765.162242:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6228, 6228] 00000080:00200000:3.0:1789760765.162262:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.162268:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.162284:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25513984 i_blocks 51209 00000020:00200000:3.0:1789760765.162290:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.162301:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.162371:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25513984, count: 4096 00000080:00200000:3.0:1789760765.162379:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25513984, bytes: 4096 00000080:00200000:3.0:1789760765.162392:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.162402:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.162404:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25513984, 25518080) 0 0 00000080:00200000:3.0:1789760765.162409:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.162426:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6229, 6229] 00020000:00200000:3.0:1789760765.162445:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25513984 chunk: [25513984, 25518080) 25518080, 4096 00020000:00200000:3.0:1789760765.162450:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.162471:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6229, 6229] 00000008:00010000:3.0:1789760765.162498:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.162504:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760765.162516:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25513984, 25518080) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.162531:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25513984=0x1855000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.162591:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.162594:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25518080 00000020:00010000:3.0:1789760765.162603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.162607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.162610:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.162613:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.162616:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.162625:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.162632:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.162655:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760765.162797:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 25513984<>25518080 00000100:00100000:2.0:1789760765.164092:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495879552:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760765.234250:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973546 gen 2 00000008:00010000:2.0:1789760765.234301:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760765.234311:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.234323:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.234344:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760765.234374:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495879552:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760765.234377:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6229, 6229] 00000080:00200000:3.0:1789760765.234398:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.234405:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.234539:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25518080 i_blocks 51209 00000020:00200000:3.0:1789760765.234544:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.234554:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.234589:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25518080, count: 4096 00000080:00200000:3.0:1789760765.234597:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25518080, bytes: 4096 00000080:00200000:3.0:1789760765.234607:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.234616:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.234618:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25518080, 25522176) 0 0 00000080:00200000:3.0:1789760765.234622:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.234640:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6230, 6230] 00020000:00200000:3.0:1789760765.234649:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25518080 chunk: [25518080, 25522176) 25522176, 4096 00020000:00200000:3.0:1789760765.234653:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.234674:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6230, 6230] 00000008:00010000:3.0:1789760765.234694:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.234699:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760765.234712:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25518080, 25522176) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.234725:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25518080=0x1856000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.234791:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.234794:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25522176 00000020:00010000:3.0:1789760765.234801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.234804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.234807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.234811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.234814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.234822:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.234829:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.234845:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760765.235104:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 25518080<>25522176 00000100:00100000:2.0:1789760765.235273:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495879680:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760765.281540:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973547 gen 2 00000008:00010000:2.0:1789760765.281594:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760765.281603:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.281615:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.281680:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760765.281711:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495879680:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760765.281902:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6230, 6230] 00000080:00200000:3.0:1789760765.281919:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.281926:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.282003:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25522176 i_blocks 51209 00000020:00200000:3.0:1789760765.282008:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.282017:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.282051:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25522176, count: 4096 00000080:00200000:3.0:1789760765.282058:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25522176, bytes: 4096 00000080:00200000:3.0:1789760765.282067:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.282076:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.282078:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25522176, 25526272) 0 0 00000080:00200000:3.0:1789760765.282083:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.282099:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6231, 6231] 00020000:00200000:3.0:1789760765.282106:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25522176 chunk: [25522176, 25526272) 25526272, 4096 00020000:00200000:3.0:1789760765.282111:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.282129:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6231, 6231] 00000008:00010000:3.0:1789760765.282146:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.282151:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760765.282161:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25522176, 25526272) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.282339:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25522176=0x1857000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.282400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.282404:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25526272 00000020:00010000:3.0:1789760765.282480:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.282485:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.282488:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.282491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.282494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.282503:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.282509:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.282528:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760765.282649:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 25522176<>25526272 00000100:00100000:2.0:1789760765.282718:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495879808:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760765.332882:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973548 gen 2 00000008:00010000:2.0:1789760765.332943:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760765.332952:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.332964:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.332986:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760765.333017:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495879808:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.333242:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6231, 6231] 00000080:00200000:0.0:1789760765.333261:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.333267:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.333284:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25526272 i_blocks 51209 00000020:00200000:0.0:1789760765.333288:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.333297:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.333331:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25526272, count: 4096 00000080:00200000:0.0:1789760765.333339:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25526272, bytes: 4096 00000080:00200000:0.0:1789760765.333348:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.333357:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.333359:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25526272, 25530368) 0 0 00000080:00200000:0.0:1789760765.333364:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.333381:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6232, 6232] 00020000:00200000:0.0:1789760765.333390:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25526272 chunk: [25526272, 25530368) 25530368, 4096 00020000:00200000:0.0:1789760765.333395:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.333416:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6232, 6232] 00000008:00010000:0.0:1789760765.333448:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.333454:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.333466:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25526272, 25530368) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.333479:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25526272=0x1858000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.333533:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.333535:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25530368 00000020:00010000:0.0:1789760765.333544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.333547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.333550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.333554:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.333557:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.333565:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.333572:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.333587:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760765.333716:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 25526272<>25530368 00000100:00100000:1.0:1789760765.333824:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760765.333853:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495879936:192.168.203.109@tcp:4:dd.0 00010000:00010000:1.0:1789760765.387622:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760765.423779:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973549 gen 2 00000008:00010000:1.0:1789760765.423818:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760765.423824:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.423834:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.423858:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760765.423885:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495879936:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.424196:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6232, 6232] 00000080:00200000:0.0:1789760765.424210:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.424226:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.424239:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25530368 i_blocks 51209 00000020:00200000:0.0:1789760765.424243:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.424252:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.424278:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25530368, count: 4096 00000080:00200000:0.0:1789760765.424284:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25530368, bytes: 4096 00000080:00200000:0.0:1789760765.424295:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.424302:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.424304:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25530368, 25534464) 0 0 00000080:00200000:0.0:1789760765.424308:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.424320:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6233, 6233] 00020000:00200000:0.0:1789760765.424327:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25530368 chunk: [25530368, 25534464) 25534464, 4096 00020000:00200000:0.0:1789760765.424331:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.424359:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6233, 6233] 00000008:00010000:0.0:1789760765.424374:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.424380:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.424387:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25530368, 25534464) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.424398:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25530368=0x1859000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.424453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.424456:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25534464 00000020:00010000:0.0:1789760765.424462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.424465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.424468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.424471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.424474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.424479:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.424484:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.424499:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760765.424773:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 25530368<>25534464 00000100:00100000:1.0:1789760765.424899:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880064:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.483587:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973550 gen 2 00000008:00010000:1.0:1789760765.483622:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760765.483628:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.483637:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.483652:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760765.483674:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880064:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.483870:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6233, 6233] 00000080:00200000:0.0:1789760765.483883:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.483888:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.483899:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25534464 i_blocks 51209 00000020:00200000:0.0:1789760765.483903:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.483910:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.483935:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25534464, count: 4096 00000080:00200000:0.0:1789760765.483941:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25534464, bytes: 4096 00000080:00200000:0.0:1789760765.483949:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.483955:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.483958:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25534464, 25538560) 0 0 00000080:00200000:0.0:1789760765.483961:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.483974:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6234, 6234] 00020000:00200000:0.0:1789760765.483981:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25534464 chunk: [25534464, 25538560) 25538560, 4096 00020000:00200000:0.0:1789760765.483985:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.484000:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6234, 6234] 00000008:00010000:0.0:1789760765.484014:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.484020:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.484029:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25534464, 25538560) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.484040:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25534464=0x185a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.484088:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.484091:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25538560 00000020:00010000:0.0:1789760765.484098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.484101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.484104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.484107:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.484110:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.484116:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.484122:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.484135:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760765.484695:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 25534464<>25538560 00000100:00100000:1.0:1789760765.485294:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760765.485314:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880192:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.521123:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973551 gen 2 00000008:00010000:1.0:1789760765.521159:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760765.521165:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.521208:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.521225:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760765.521249:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880192:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.521348:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6234, 6234] 00000080:00200000:0.0:1789760765.521361:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.521366:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.521379:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25538560 i_blocks 51209 00000020:00200000:0.0:1789760765.521382:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.521389:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.521415:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25538560, count: 4096 00000080:00200000:0.0:1789760765.521421:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25538560, bytes: 4096 00000080:00200000:0.0:1789760765.521436:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.521443:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.521445:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25538560, 25542656) 0 0 00000080:00200000:0.0:1789760765.521449:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.521462:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6235, 6235] 00020000:00200000:0.0:1789760765.521612:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25538560 chunk: [25538560, 25542656) 25542656, 4096 00020000:00200000:0.0:1789760765.521617:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.521634:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6235, 6235] 00000008:00010000:0.0:1789760765.521648:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.521653:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.521662:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25538560, 25542656) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.521672:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25538560=0x185b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.521717:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.521720:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25542656 00000020:00010000:0.0:1789760765.521726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.521730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.521733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.521736:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.521738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.521757:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.521763:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.521780:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760765.521876:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 25538560<>25542656 00000100:00100000:1.0:1789760765.522310:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880320:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.597284:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973552 gen 2 00000008:00010000:1.0:1789760765.597335:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760765.597344:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.597355:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.597377:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000080:00200000:0.0:1789760765.597498:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6235, 6235] 00000080:00200000:0.0:1789760765.597516:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.597522:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.597539:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25542656 i_blocks 51209 00000020:00200000:0.0:1789760765.597543:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.597552:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.597586:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25542656, count: 4096 00000080:00200000:0.0:1789760765.597594:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25542656, bytes: 4096 00000080:00200000:0.0:1789760765.597604:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.597612:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.597615:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25542656, 25546752) 0 0 00000080:00200000:0.0:1789760765.597619:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.597635:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6236, 6236] 00020000:00200000:0.0:1789760765.597643:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25542656 chunk: [25542656, 25546752) 25546752, 4096 00020000:00200000:0.0:1789760765.597647:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.597668:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6236, 6236] 00000008:00010000:0.0:1789760765.597687:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.597692:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.597705:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25542656, 25546752) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.597719:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25542656=0x185c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.597843:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.597846:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25546752 00000020:00010000:0.0:1789760765.598043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.598048:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.598051:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.598055:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.598058:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.598065:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.598073:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760765.598136:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880320:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760765.598215:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:1.0:1789760765.598839:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25542656<>25546752 00000100:00100000:1.0:1789760765.599242:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495880448:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760765.641386:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760765.641402:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.641424:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.641454:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760765.641485:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495880448:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.641791:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6236, 6236] 00000080:00200000:0.0:1789760765.641811:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.641818:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.641834:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25546752 i_blocks 51209 00000020:00200000:0.0:1789760765.641838:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.641847:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.641881:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25546752, count: 4096 00000080:00200000:0.0:1789760765.641889:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25546752, bytes: 4096 00000080:00200000:0.0:1789760765.641901:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.641910:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.641912:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25546752, 25550848) 0 0 00000080:00200000:0.0:1789760765.641916:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.641933:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6237, 6237] 00020000:00200000:0.0:1789760765.641941:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25546752 chunk: [25546752, 25550848) 25550848, 4096 00020000:00200000:0.0:1789760765.641945:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.641966:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6237, 6237] 00000008:00010000:0.0:1789760765.641984:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.641990:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.642002:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25546752, 25550848) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.642017:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25546752=0x185d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.642310:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.642314:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25550848 00000020:00010000:0.0:1789760765.642324:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.642328:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.642331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.642334:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.642337:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.642346:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.642353:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.642454:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:1.0:1789760765.642983:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 25546752<>25550848 00000100:00100000:1.0:1789760765.643048:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760765.643086:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495880576:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.679565:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973553 gen 2 00000100:00100000:1.0:1789760765.679606:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504dc00 x1876699495880576/t4294973554(4294973554) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760781 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760765.679655:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760765.679662:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.679673:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.679739:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760765.679765:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495880576:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.679976:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6237, 6237] 00000080:00200000:0.0:1789760765.679993:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.679999:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.680015:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25550848 i_blocks 51209 00000020:00200000:0.0:1789760765.680018:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.680028:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.680064:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25550848, count: 4096 00000080:00200000:0.0:1789760765.680071:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25550848, bytes: 4096 00000080:00200000:0.0:1789760765.680080:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.680090:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.680092:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25550848, 25554944) 0 0 00000080:00200000:0.0:1789760765.680097:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.680112:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6238, 6238] 00020000:00200000:0.0:1789760765.680121:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25550848 chunk: [25550848, 25554944) 25554944, 4096 00020000:00200000:0.0:1789760765.680125:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.680142:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6238, 6238] 00000008:00010000:0.0:1789760765.680159:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.680164:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.680437:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25550848, 25554944) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.680453:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25550848=0x185e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.680510:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.680513:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25554944 00000020:00010000:0.0:1789760765.680521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.680524:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.680528:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.680531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.680534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.680542:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.680548:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.680565:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760765.680686:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 25550848<>25554944 00000100:00100000:1.0:1789760765.680756:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495880704:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.723065:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973555 gen 2 00000008:00010000:1.0:1789760765.723231:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760765.723240:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.723251:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.723316:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760765.723345:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495880704:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.724343:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6238, 6238] 00000080:00200000:0.0:1789760765.724365:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.724372:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.724388:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25554944 i_blocks 51209 00000020:00200000:0.0:1789760765.724393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.724405:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.726269:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25554944, count: 4096 00000080:00200000:0.0:1789760765.726288:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25554944, bytes: 4096 00000080:00200000:0.0:1789760765.726305:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.726315:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.726317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25554944, 25559040) 0 0 00000080:00200000:0.0:1789760765.726323:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.726341:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6239, 6239] 00020000:00200000:0.0:1789760765.726350:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25554944 chunk: [25554944, 25559040) 25559040, 4096 00020000:00200000:0.0:1789760765.726355:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.726377:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6239, 6239] 00000008:00010000:0.0:1789760765.726396:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.726402:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.726413:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25554944, 25559040) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.726440:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25554944=0x185f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.726512:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.726516:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25559040 00000020:00010000:0.0:1789760765.726524:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.726527:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.726599:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.726603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.726605:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.726616:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.726623:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.726654:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760765.726802:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 25554944<>25559040 00000100:00100000:1.0:1789760765.726924:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880832:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.773239:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973556 gen 2 00000008:00010000:1.0:1789760765.773290:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760765.773300:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.773310:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.773331:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760765.773362:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880832:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.773479:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6239, 6239] 00000080:00200000:0.0:1789760765.773496:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.773502:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.773613:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25559040 i_blocks 51209 00000020:00200000:0.0:1789760765.773618:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.773628:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.773664:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25559040, count: 4096 00000080:00200000:0.0:1789760765.773672:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25559040, bytes: 4096 00000080:00200000:0.0:1789760765.773683:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.773692:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.773694:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25559040, 25563136) 0 0 00000080:00200000:0.0:1789760765.773699:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.773715:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6240, 6240] 00020000:00200000:0.0:1789760765.773723:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25559040 chunk: [25559040, 25563136) 25563136, 4096 00020000:00200000:0.0:1789760765.773728:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.773747:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6240, 6240] 00000008:00010000:0.0:1789760765.773765:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.773771:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.773782:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25559040, 25563136) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.773797:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25559040=0x1860000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.773855:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.773858:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25563136 00000020:00010000:0.0:1789760765.773867:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.773871:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.773874:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.773877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.773880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.773887:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.773894:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.773910:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760765.774102:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 25559040<>25563136 00000100:00100000:1.0:1789760765.774234:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760765.774251:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880960:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.811023:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973557 gen 2 00000008:00010000:1.0:1789760765.811150:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760765.811160:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.811211:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.811233:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760765.811261:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495880960:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.811499:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6240, 6240] 00000080:00200000:0.0:1789760765.811517:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.811523:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.811540:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25563136 i_blocks 51209 00000020:00200000:0.0:1789760765.811543:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.811553:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.811587:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25563136, count: 4096 00000080:00200000:0.0:1789760765.811595:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25563136, bytes: 4096 00000080:00200000:0.0:1789760765.811605:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.811614:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.811617:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25563136, 25567232) 0 0 00000080:00200000:0.0:1789760765.811622:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.811638:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6241, 6241] 00020000:00200000:0.0:1789760765.811647:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25563136 chunk: [25563136, 25567232) 25567232, 4096 00020000:00200000:0.0:1789760765.811652:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.811672:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6241, 6241] 00000008:00010000:0.0:1789760765.811689:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.811700:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.811713:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25563136, 25567232) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.811729:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25563136=0x1861000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.811787:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.811790:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25567232 00000020:00010000:0.0:1789760765.811797:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.811801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.811804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.811808:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.811811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.811817:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.811824:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.811840:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760765.812104:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 25563136<>25567232 00000100:00100000:1.0:1789760765.812266:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495881088:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760765.836505:0:468:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1789760765.836526:0:468:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760765.836602:0:468:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1789760765.857702:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973558 gen 2 00000008:00010000:1.0:1789760765.857762:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760765.857771:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.857782:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.857804:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760765.857834:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495881088:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.857937:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6241, 6241] 00000080:00200000:0.0:1789760765.857955:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.857961:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.857977:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25567232 i_blocks 51209 00000020:00200000:0.0:1789760765.857981:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.857990:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.858025:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25567232, count: 4096 00000080:00200000:0.0:1789760765.858034:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25567232, bytes: 4096 00000080:00200000:0.0:1789760765.858042:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.858052:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.858054:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25567232, 25571328) 0 0 00000080:00200000:0.0:1789760765.858059:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.858076:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6242, 6242] 00020000:00200000:0.0:1789760765.858084:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25567232 chunk: [25567232, 25571328) 25571328, 4096 00020000:00200000:0.0:1789760765.858089:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.858107:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6242, 6242] 00000008:00010000:0.0:1789760765.858124:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.858129:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.858140:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25567232, 25571328) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.858154:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25567232=0x1862000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.858304:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.858308:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25571328 00000020:00010000:0.0:1789760765.858316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.858320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.858323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.858326:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.858329:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.858336:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.858648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.858766:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:1.0:1789760765.858966:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 25567232<>25571328 00000100:00100000:1.0:1789760765.859015:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495881216:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760765.904378:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760765.904404:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.904417:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.904444:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760765.904474:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495881216:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.904635:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6242, 6242] 00000080:00200000:0.0:1789760765.904659:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.904665:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.904681:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25571328 i_blocks 51209 00000020:00200000:0.0:1789760765.904685:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.904694:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.904725:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25571328, count: 4096 00000080:00200000:0.0:1789760765.904733:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25571328, bytes: 4096 00000080:00200000:0.0:1789760765.904743:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.904751:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.904754:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25571328, 25575424) 0 0 00000080:00200000:0.0:1789760765.904757:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.904773:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6243, 6243] 00020000:00200000:0.0:1789760765.904781:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25571328 chunk: [25571328, 25575424) 25575424, 4096 00020000:00200000:0.0:1789760765.904786:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.904806:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6243, 6243] 00000008:00010000:0.0:1789760765.904834:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.904840:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.904852:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25571328, 25575424) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.904867:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25571328=0x1863000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.904927:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.904931:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25575424 00000020:00010000:0.0:1789760765.904940:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.904944:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.904947:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.904951:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.904954:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.904961:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.904968:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.904984:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760765.905610:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 25571328<>25575424 00000100:00100000:1.0:1789760765.905793:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495881344:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.941399:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973560 gen 2 00000008:00010000:1.0:1789760765.941471:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760765.941480:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.941490:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.941513:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760765.941540:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495881344:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760765.941716:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6243, 6243] 00000080:00200000:0.0:1789760765.941733:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.941739:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.941755:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25575424 i_blocks 51209 00000020:00200000:0.0:1789760765.941760:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.941769:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.941803:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25575424, count: 4096 00000080:00200000:0.0:1789760765.941810:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25575424, bytes: 4096 00000080:00200000:0.0:1789760765.941820:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.941829:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.941832:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25575424, 25579520) 0 0 00000080:00200000:0.0:1789760765.941836:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.941852:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6244, 6244] 00020000:00200000:0.0:1789760765.941860:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25575424 chunk: [25575424, 25579520) 25579520, 4096 00020000:00200000:0.0:1789760765.941865:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.941883:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6244, 6244] 00000008:00010000:0.0:1789760765.942031:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.942037:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760765.942049:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25575424, 25579520) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.942063:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25575424=0x1864000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.942120:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.942123:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25579520 00000020:00010000:0.0:1789760765.942132:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.942135:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.942138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.942141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.942144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.942151:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.942157:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.942434:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760765.942628:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 25575424<>25579520 00000100:00100000:0.0:1789760765.942672:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760765.942689:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495881472:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760765.994749:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973561 gen 2 00000008:00010000:1.0:1789760765.994795:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760765.994803:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.994814:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.994841:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760765.994869:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495881472:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760765.994985:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6244, 6244] 00000080:00200000:2.0:1789760765.995002:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760765.995008:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760765.995023:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25579520 i_blocks 51209 00000020:00200000:2.0:1789760765.995027:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760765.995037:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760765.995073:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25579520, count: 4096 00000080:00200000:2.0:1789760765.995080:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25579520, bytes: 4096 00000080:00200000:2.0:1789760765.995091:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760765.995100:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760765.995103:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25579520, 25583616) 0 0 00000080:00200000:2.0:1789760765.995107:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760765.995123:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6245, 6245] 00020000:00200000:2.0:1789760765.995132:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25579520 chunk: [25579520, 25583616) 25583616, 4096 00020000:00200000:2.0:1789760765.995136:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760765.995156:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6245, 6245] 00000008:00010000:2.0:1789760765.995208:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760765.995215:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760765.995227:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25579520, 25583616) append 0 append lockpos 0 00000080:00200000:2.0:1789760765.995243:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25579520=0x1865000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760765.995298:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760765.995301:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25583616 00000020:00010000:2.0:1789760765.995309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760765.995313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.995316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.995319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760765.995322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760765.995328:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760765.995335:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760765.995351:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760765.995486:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 25579520<>25583616 00000100:00100000:2.0:1789760765.995532:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760765.995549:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495881600:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760766.051735:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973562 gen 2 00000008:00010000:2.0:1789760766.051777:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760766.051784:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760766.051793:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760766.051819:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760766.051847:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495881600:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760766.051973:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6245, 6245] 00000080:00200000:1.0:1789760766.051999:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.052007:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.052023:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25583616 i_blocks 51209 00000020:00200000:1.0:1789760766.052027:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.052038:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.052073:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25583616, count: 4096 00000080:00200000:1.0:1789760766.052081:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25583616, bytes: 4096 00000080:00200000:1.0:1789760766.052094:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.052102:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.052105:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25583616, 25587712) 0 0 00000080:00200000:1.0:1789760766.052110:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.052127:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6246, 6246] 00020000:00200000:1.0:1789760766.052146:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25583616 chunk: [25583616, 25587712) 25587712, 4096 00020000:00200000:1.0:1789760766.052151:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.052233:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6246, 6246] 00000008:00010000:1.0:1789760766.052255:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.052261:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760766.052273:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25583616, 25587712) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.052289:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25583616=0x1866000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.052342:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.052345:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25587712 00000020:00010000:1.0:1789760766.052353:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.052357:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.052361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.052365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.052367:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.052374:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.052381:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.052394:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760766.052785:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 25583616<>25587712 00000100:00100000:1.0:1789760766.052836:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760766.052854:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495881728:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.089537:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973563 gen 2 00000008:00010000:1.0:1789760766.089589:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760766.089598:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.089610:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.089639:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760766.089669:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495881728:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760766.091059:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6246, 6246] 00000080:00200000:2.0:1789760766.091085:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.091091:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.091109:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25587712 i_blocks 51209 00000020:00200000:2.0:1789760766.091114:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.091127:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.091163:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25587712, count: 4096 00000080:00200000:2.0:1789760766.091240:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25587712, bytes: 4096 00000080:00200000:2.0:1789760766.091277:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.091287:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.091289:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25587712, 25591808) 0 0 00000080:00200000:2.0:1789760766.091294:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.091309:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6247, 6247] 00020000:00200000:2.0:1789760766.091318:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25587712 chunk: [25587712, 25591808) 25591808, 4096 00020000:00200000:2.0:1789760766.091323:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.091344:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6247, 6247] 00000008:00010000:2.0:1789760766.091361:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.091366:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760766.091378:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25587712, 25591808) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.091392:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25587712=0x1867000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.091464:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.091467:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25591808 00000020:00010000:2.0:1789760766.091475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.091479:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.091482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.091485:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.091488:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.091496:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.091502:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.091516:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760766.091650:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 25587712<>25591808 00000100:00100000:3.0:1789760766.091740:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495881856:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760766.157142:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973564 gen 2 00000008:00010000:2.0:1789760766.157224:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760766.157233:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760766.157245:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760766.157326:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760766.157354:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495881856:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760766.161690:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6247, 6247] 00000080:00200000:1.0:1789760766.161718:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.161725:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.161745:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25591808 i_blocks 51209 00000020:00200000:1.0:1789760766.161751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.161768:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.161804:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25591808, count: 4096 00000080:00200000:1.0:1789760766.161812:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25591808, bytes: 4096 00000080:00200000:1.0:1789760766.161828:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.161837:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.161839:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25591808, 25595904) 0 0 00000080:00200000:1.0:1789760766.161843:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.161861:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6248, 6248] 00020000:00200000:1.0:1789760766.161870:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25591808 chunk: [25591808, 25595904) 25595904, 4096 00020000:00200000:1.0:1789760766.161876:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.161898:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6248, 6248] 00000008:00010000:1.0:1789760766.161917:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.161923:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760766.161935:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25591808, 25595904) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.161949:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25591808=0x1868000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.162016:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.162019:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25595904 00000020:00010000:1.0:1789760766.162027:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.162031:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.162034:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.162037:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.162040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.162047:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.162054:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.162085:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760766.162251:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 25591808<>25595904 00000100:00100000:1.0:1789760766.162304:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760766.162330:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495881984:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.193038:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973565 gen 2 00000008:00010000:1.0:1789760766.193087:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760766.193096:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.193107:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.193303:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760766.193334:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495881984:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.193448:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6248, 6248] 00000080:00200000:0.0:1789760766.193466:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.193472:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.193488:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25595904 i_blocks 51209 00000020:00200000:0.0:1789760766.193492:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.193502:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.193536:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25595904, count: 4096 00000080:00200000:0.0:1789760766.193544:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25595904, bytes: 4096 00000080:00200000:0.0:1789760766.193554:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.193563:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.193566:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25595904, 25600000) 0 0 00000080:00200000:0.0:1789760766.193570:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.193588:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6249, 6249] 00020000:00200000:0.0:1789760766.193597:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25595904 chunk: [25595904, 25600000) 25600000, 4096 00020000:00200000:0.0:1789760766.193602:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.193623:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6249, 6249] 00000008:00010000:0.0:1789760766.193640:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.193645:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.193656:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25595904, 25600000) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.193670:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25595904=0x1869000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.193731:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.193734:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25600000 00000020:00010000:0.0:1789760766.193813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.193817:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.193820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.193824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.193827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.193894:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.193902:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.193922:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760766.194456:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 25595904<>25600000 00000100:00100000:1.0:1789760766.194520:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495882112:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.232603:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973566 gen 2 00000008:00010000:1.0:1789760766.232654:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760766.232662:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.232673:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.232768:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760766.232799:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495882112:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760766.239360:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6249, 6249] 00000080:00200000:2.0:1789760766.239393:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.239400:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.239431:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25600000 i_blocks 51209 00000020:00200000:2.0:1789760766.239437:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.239765:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.239800:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25600000, count: 4096 00000080:00200000:2.0:1789760766.239810:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25600000, bytes: 4096 00000080:00200000:2.0:1789760766.239825:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.239835:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.239837:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25600000, 25604096) 0 0 00000080:00200000:2.0:1789760766.239842:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.239860:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6250, 6250] 00020000:00200000:2.0:1789760766.239869:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25600000 chunk: [25600000, 25604096) 25604096, 4096 00020000:00200000:2.0:1789760766.239874:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.239897:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6250, 6250] 00000008:00010000:2.0:1789760766.239917:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.239923:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760766.239935:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25600000, 25604096) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.239949:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25600000=0x186a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.240009:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.240011:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25604096 00000020:00010000:2.0:1789760766.240019:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.240023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.240026:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.240029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.240032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.240039:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.240566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.240594:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760766.240732:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 25600000<>25604096 00000100:00100000:3.0:1789760766.240836:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495882240:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760766.283634:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973567 gen 2 00000008:00010000:3.0:1789760766.283686:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760766.283694:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.283706:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.283730:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760766.283757:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495882240:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760766.283868:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6250, 6250] 00000080:00200000:2.0:1789760766.283886:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.283892:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.283910:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25604096 i_blocks 51209 00000020:00200000:2.0:1789760766.283914:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.283924:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.283957:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25604096, count: 4096 00000080:00200000:2.0:1789760766.283965:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25604096, bytes: 4096 00000080:00200000:2.0:1789760766.283975:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.283984:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.283987:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25604096, 25608192) 0 0 00000080:00200000:2.0:1789760766.283991:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.284010:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6251, 6251] 00020000:00200000:2.0:1789760766.284018:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25604096 chunk: [25604096, 25608192) 25608192, 4096 00020000:00200000:2.0:1789760766.284023:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.284043:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6251, 6251] 00000008:00010000:2.0:1789760766.284062:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.284068:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760766.284081:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25604096, 25608192) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.284096:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25604096=0x186b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.284164:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.284302:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25608192 00000020:00010000:2.0:1789760766.284313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.284318:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.284321:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.284325:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.284328:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.284336:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.284344:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.284363:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760766.284611:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 25604096<>25608192 00000100:00100000:3.0:1789760766.284676:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495882368:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760766.335254:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973568 gen 2 00000008:00010000:3.0:1789760766.335304:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760766.335313:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.335326:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.335339:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760766.335369:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495882368:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.335574:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6251, 6251] 00000080:00200000:0.0:1789760766.335593:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.335599:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.335616:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25608192 i_blocks 51209 00000020:00200000:0.0:1789760766.335620:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.335630:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.335665:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25608192, count: 4096 00000080:00200000:0.0:1789760766.335673:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25608192, bytes: 4096 00000080:00200000:0.0:1789760766.335684:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.335693:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.335695:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25608192, 25612288) 0 0 00000080:00200000:0.0:1789760766.335700:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.335717:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6252, 6252] 00020000:00200000:0.0:1789760766.335726:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25608192 chunk: [25608192, 25612288) 25612288, 4096 00020000:00200000:0.0:1789760766.335731:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.335753:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6252, 6252] 00000008:00010000:0.0:1789760766.335770:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.335775:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.335792:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25608192, 25612288) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.335807:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25608192=0x186c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.335868:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.335871:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25612288 00000020:00010000:0.0:1789760766.335880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.335883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.335887:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.335890:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.335892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.335899:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.335906:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.335922:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760766.336045:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 25608192<>25612288 00000100:00100000:1.0:1789760766.336161:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760766.336214:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495882496:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.379463:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973569 gen 2 00000008:00010000:1.0:1789760766.379515:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760766.379524:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.379534:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.379557:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760766.379585:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495882496:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.379698:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6252, 6252] 00000080:00200000:0.0:1789760766.379715:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.379721:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.379737:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25612288 i_blocks 51209 00000020:00200000:0.0:1789760766.379751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.379761:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.379796:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25612288, count: 4096 00000080:00200000:0.0:1789760766.379805:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25612288, bytes: 4096 00000080:00200000:0.0:1789760766.379814:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.379823:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.379825:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25612288, 25616384) 0 0 00000080:00200000:0.0:1789760766.379830:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.379846:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6253, 6253] 00020000:00200000:0.0:1789760766.379854:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25612288 chunk: [25612288, 25616384) 25616384, 4096 00020000:00200000:0.0:1789760766.379859:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.379878:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6253, 6253] 00000008:00010000:0.0:1789760766.379895:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.379901:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.379912:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25612288, 25616384) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.379928:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25612288=0x186d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.379986:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.379989:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25616384 00000020:00010000:0.0:1789760766.379997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.380001:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.380004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.380008:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.380011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.380019:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.380026:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.380231:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760766.380356:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 25612288<>25616384 00000100:00100000:1.0:1789760766.380459:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495882624:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760766.417801:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760766.434483:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760766.434499:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.434510:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.434596:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760766.434628:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495882624:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.436264:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6253, 6253] 00000080:00200000:0.0:1789760766.436288:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.436295:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.436312:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25616384 i_blocks 51209 00000020:00200000:0.0:1789760766.436317:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.436332:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.436366:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25616384, count: 4096 00000080:00200000:0.0:1789760766.436374:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25616384, bytes: 4096 00000080:00200000:0.0:1789760766.436389:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.436399:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.436401:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25616384, 25620480) 0 0 00000080:00200000:0.0:1789760766.436406:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.438355:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6254, 6254] 00020000:00200000:0.0:1789760766.438376:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25616384 chunk: [25616384, 25620480) 25620480, 4096 00020000:00200000:0.0:1789760766.438382:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.438412:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6254, 6254] 00000008:00010000:0.0:1789760766.438446:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.438453:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.438467:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25616384, 25620480) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.438485:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25616384=0x186e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.438725:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.438730:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25620480 00000020:00010000:0.0:1789760766.438739:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.438742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.438745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.438749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.438751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.438764:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.438771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.438803:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760766.438933:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 25616384<>25620480 00000100:00100000:1.0:1789760766.439025:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495882752:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.486042:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973570 gen 2 00000100:00100000:1.0:1789760766.486087:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211836d80 x1876699495882752/t4294973571(4294973571) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760782 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760766.486137:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760766.486145:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.486154:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.486206:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760766.486229:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495882752:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.486371:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6254, 6254] 00000080:00200000:0.0:1789760766.486388:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.486394:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.486408:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25620480 i_blocks 51209 00000020:00200000:0.0:1789760766.486412:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.486430:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.486465:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25620480, count: 4096 00000080:00200000:0.0:1789760766.486472:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25620480, bytes: 4096 00000080:00200000:0.0:1789760766.486483:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.486493:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.486495:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25620480, 25624576) 0 0 00000080:00200000:0.0:1789760766.486499:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.486526:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6255, 6255] 00020000:00200000:0.0:1789760766.486535:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25620480 chunk: [25620480, 25624576) 25624576, 4096 00020000:00200000:0.0:1789760766.486539:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.486569:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6255, 6255] 00000008:00010000:0.0:1789760766.486588:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.486594:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.486605:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25620480, 25624576) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.486620:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25620480=0x186f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.486683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.486686:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25624576 00000020:00010000:0.0:1789760766.486694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.486698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.486701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.486705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.486708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.486716:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.486723:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.486749:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760766.486870:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 25620480<>25624576 00000100:00100000:1.0:1789760766.486936:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495882880:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.520199:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973572 gen 2 00000008:00010000:1.0:1789760766.520375:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760766.520383:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.520393:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.520413:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760766.520452:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495882880:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.522091:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6255, 6255] 00000080:00200000:0.0:1789760766.522117:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.522124:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.522143:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25624576 i_blocks 51209 00000020:00200000:0.0:1789760766.522148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.522161:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.522213:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25624576, count: 4096 00000080:00200000:0.0:1789760766.522222:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25624576, bytes: 4096 00000080:00200000:0.0:1789760766.522240:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.522404:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.522406:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25624576, 25628672) 0 0 00000080:00200000:0.0:1789760766.522412:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.522438:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6256, 6256] 00020000:00200000:0.0:1789760766.522447:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25624576 chunk: [25624576, 25628672) 25628672, 4096 00020000:00200000:0.0:1789760766.522451:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.522473:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6256, 6256] 00000008:00010000:0.0:1789760766.522491:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.522497:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.522509:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25624576, 25628672) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.522526:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25624576=0x1870000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.522594:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.522598:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25628672 00000020:00010000:0.0:1789760766.522606:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.522609:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.522613:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.522617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.522619:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.522627:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.522635:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.522658:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760766.522794:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 25624576<>25628672 00000100:00100000:1.0:1789760766.524338:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495883008:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.561590:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973573 gen 2 00000008:00010000:1.0:1789760766.561641:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760766.561650:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.561661:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.561681:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760766.561710:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495883008:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.561873:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6256, 6256] 00000080:00200000:0.0:1789760766.561888:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.561894:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.561908:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25628672 i_blocks 51209 00000020:00200000:0.0:1789760766.561912:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.561922:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.561955:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25628672, count: 4096 00000080:00200000:0.0:1789760766.561962:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25628672, bytes: 4096 00000080:00200000:0.0:1789760766.561971:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.561980:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.561982:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25628672, 25632768) 0 0 00000080:00200000:0.0:1789760766.561986:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.562003:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6257, 6257] 00020000:00200000:0.0:1789760766.562012:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25628672 chunk: [25628672, 25632768) 25632768, 4096 00020000:00200000:0.0:1789760766.562016:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.562037:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6257, 6257] 00000008:00010000:0.0:1789760766.562125:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.562132:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.562144:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25628672, 25632768) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.562159:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25628672=0x1871000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.562326:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.562330:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25632768 00000020:00010000:0.0:1789760766.562338:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.562342:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.562345:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.562349:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.562352:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.562360:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.562367:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.562384:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760766.562583:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 25628672<>25632768 00000100:00100000:1.0:1789760766.562876:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495883136:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.595821:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973574 gen 2 00000008:00010000:1.0:1789760766.595875:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760766.595883:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.595894:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.595915:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760766.595948:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495883136:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.596145:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6257, 6257] 00000080:00200000:0.0:1789760766.596160:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.596165:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.596212:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25632768 i_blocks 51209 00000020:00200000:0.0:1789760766.596215:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.596223:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.596254:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25632768, count: 4096 00000080:00200000:0.0:1789760766.596260:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25632768, bytes: 4096 00000080:00200000:0.0:1789760766.596268:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.596275:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.596277:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25632768, 25636864) 0 0 00000080:00200000:0.0:1789760766.596281:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.596295:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6258, 6258] 00020000:00200000:0.0:1789760766.596302:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25632768 chunk: [25632768, 25636864) 25636864, 4096 00020000:00200000:0.0:1789760766.596305:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.596322:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6258, 6258] 00000008:00010000:0.0:1789760766.596336:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.596340:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.596350:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25632768, 25636864) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.596362:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25632768=0x1872000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.596411:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.596414:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25636864 00000020:00010000:0.0:1789760766.596430:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.596434:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.596437:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.596440:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.596443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.596450:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.596457:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.596474:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760766.596588:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 25632768<>25636864 00000100:00100000:1.0:1789760766.598098:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760766.598134:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495883264:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.627017:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973575 gen 2 00000008:00010000:1.0:1789760766.627069:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760766.627078:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.627089:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.627112:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760766.627143:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495883264:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.627298:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6258, 6258] 00000080:00200000:0.0:1789760766.627316:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.627323:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.627340:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25636864 i_blocks 51209 00000020:00200000:0.0:1789760766.627344:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.627353:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.627386:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25636864, count: 4096 00000080:00200000:0.0:1789760766.627395:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25636864, bytes: 4096 00000080:00200000:0.0:1789760766.627416:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.627434:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.627437:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25636864, 25640960) 0 0 00000080:00200000:0.0:1789760766.627441:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.627458:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6259, 6259] 00020000:00200000:0.0:1789760766.627467:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25636864 chunk: [25636864, 25640960) 25640960, 4096 00020000:00200000:0.0:1789760766.627472:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.627492:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6259, 6259] 00000008:00010000:0.0:1789760766.627510:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.627516:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.627528:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25636864, 25640960) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.627654:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25636864=0x1873000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.627710:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.627714:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25640960 00000020:00010000:0.0:1789760766.627723:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.627727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.627730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.627734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.627737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.627809:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.627817:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.627893:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:1.0:1789760766.628024:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 25636864<>25640960 00000100:00100000:1.0:1789760766.628082:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495883392:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.659610:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973576 gen 2 00000008:00010000:1.0:1789760766.659660:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760766.659668:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.659678:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.659722:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000080:00200000:0.0:1789760766.659725:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6259, 6259] 00000100:00100000:1.0:1789760766.659757:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495883392:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.659803:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.659810:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.659826:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25640960 i_blocks 51209 00000020:00200000:0.0:1789760766.659830:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.659840:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.659874:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25640960, count: 4096 00000080:00200000:0.0:1789760766.659880:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25640960, bytes: 4096 00000080:00200000:0.0:1789760766.659890:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.659898:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.659901:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25640960, 25645056) 0 0 00000080:00200000:0.0:1789760766.659905:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.659923:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6260, 6260] 00020000:00200000:0.0:1789760766.659933:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25640960 chunk: [25640960, 25645056) 25645056, 4096 00020000:00200000:0.0:1789760766.659938:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.659961:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6260, 6260] 00000008:00010000:0.0:1789760766.659980:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.659986:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.659998:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25640960, 25645056) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.660013:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25640960=0x1874000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.660073:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.660077:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25645056 00000020:00010000:0.0:1789760766.660085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.660088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.660091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.660095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.660098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.660105:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.660112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.660128:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760766.660276:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 25640960<>25645056 00000100:00100000:1.0:1789760766.660372:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495883520:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.692805:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973577 gen 2 00000008:00010000:1.0:1789760766.692855:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760766.692864:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.692875:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.692895:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000080:00200000:0.0:1789760766.692924:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6260, 6260] 00000100:00100000:1.0:1789760766.692924:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495883520:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.692941:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.692947:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.692962:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25645056 i_blocks 51209 00000020:00200000:0.0:1789760766.692966:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.692976:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.693012:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25645056, count: 4096 00000080:00200000:0.0:1789760766.693020:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25645056, bytes: 4096 00000080:00200000:0.0:1789760766.693029:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.693038:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.693040:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25645056, 25649152) 0 0 00000080:00200000:0.0:1789760766.693045:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.693062:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6261, 6261] 00020000:00200000:0.0:1789760766.693074:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25645056 chunk: [25645056, 25649152) 25649152, 4096 00020000:00200000:0.0:1789760766.693079:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.693100:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6261, 6261] 00000008:00010000:0.0:1789760766.693122:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.693129:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.693141:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25645056, 25649152) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.693155:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25645056=0x1875000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.693240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.693243:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25649152 00000020:00010000:0.0:1789760766.693252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.693255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.693259:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.693263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.693265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.693272:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.693278:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.693294:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760766.693412:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 25645056<>25649152 00000100:00100000:1.0:1789760766.693494:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760766.693509:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495883648:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.724009:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973578 gen 2 00000008:00010000:1.0:1789760766.724066:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760766.724073:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.724084:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.724148:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760766.724230:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495883648:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.724617:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6261, 6261] 00000080:00200000:0.0:1789760766.724635:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.724642:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.724721:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25649152 i_blocks 51209 00000020:00200000:0.0:1789760766.724727:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.724736:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.724771:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25649152, count: 4096 00000080:00200000:0.0:1789760766.724778:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25649152, bytes: 4096 00000080:00200000:0.0:1789760766.724788:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.724798:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.724800:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25649152, 25653248) 0 0 00000080:00200000:0.0:1789760766.724805:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.724823:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6262, 6262] 00020000:00200000:0.0:1789760766.724830:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25649152 chunk: [25649152, 25653248) 25653248, 4096 00020000:00200000:0.0:1789760766.724963:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.724985:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6262, 6262] 00000008:00010000:0.0:1789760766.725004:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.725010:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.725024:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25649152, 25653248) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.725037:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25649152=0x1876000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.725100:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.725104:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25653248 00000020:00010000:0.0:1789760766.725111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.725115:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.725117:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.725121:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.725124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.725131:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.725138:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.725158:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760766.725367:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 25649152<>25653248 00000100:00100000:1.0:1789760766.725458:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495883776:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.769867:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973579 gen 2 00000008:00010000:1.0:1789760766.769920:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760766.769930:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.769942:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.769972:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760766.770002:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495883776:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.770880:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6262, 6262] 00000080:00200000:0.0:1789760766.770900:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.770907:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.770921:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25653248 i_blocks 51209 00000020:00200000:0.0:1789760766.770926:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.770936:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.770969:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25653248, count: 4096 00000080:00200000:0.0:1789760766.770976:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25653248, bytes: 4096 00000080:00200000:0.0:1789760766.770991:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.771000:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.771002:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25653248, 25657344) 0 0 00000080:00200000:0.0:1789760766.771007:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.771024:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6263, 6263] 00020000:00200000:0.0:1789760766.771033:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25653248 chunk: [25653248, 25657344) 25657344, 4096 00020000:00200000:0.0:1789760766.771038:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.771059:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6263, 6263] 00000008:00010000:0.0:1789760766.771077:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.771084:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.771095:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25653248, 25657344) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.771110:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25653248=0x1877000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.771217:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.771221:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25657344 00000020:00010000:0.0:1789760766.771229:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.771233:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.771236:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.771239:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.771242:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.771250:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.771257:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.771274:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760766.771400:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25653248<>25657344 00000100:00100000:1.0:1789760766.772264:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495883904:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.807252:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973580 gen 2 00000008:00010000:1.0:1789760766.807303:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760766.807312:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.807324:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.807343:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760766.807382:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495883904:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.807555:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6263, 6263] 00000080:00200000:0.0:1789760766.807574:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.807581:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.807597:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25657344 i_blocks 51209 00000020:00200000:0.0:1789760766.807601:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.807609:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.807645:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25657344, count: 4096 00000080:00200000:0.0:1789760766.807653:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25657344, bytes: 4096 00000080:00200000:0.0:1789760766.807663:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.807672:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.807674:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25657344, 25661440) 0 0 00000080:00200000:0.0:1789760766.807679:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.807697:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6264, 6264] 00020000:00200000:0.0:1789760766.807705:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25657344 chunk: [25657344, 25661440) 25661440, 4096 00020000:00200000:0.0:1789760766.807710:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.807732:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6264, 6264] 00000008:00010000:0.0:1789760766.807978:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.807984:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.807996:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25657344, 25661440) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.808011:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25657344=0x1878000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.808072:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.808076:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25661440 00000020:00010000:0.0:1789760766.808084:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.808088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.808091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.808095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.808098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.808106:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.808112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.808129:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760766.808301:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 25657344<>25661440 00000100:00100000:1.0:1789760766.808489:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760766.808507:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884032:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.845951:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973581 gen 2 00000008:00010000:1.0:1789760766.846000:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760766.846008:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.846019:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.846042:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760766.846073:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884032:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.846204:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6264, 6264] 00000080:00200000:0.0:1789760766.846220:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.846227:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.846252:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25661440 i_blocks 51209 00000020:00200000:0.0:1789760766.846256:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.846265:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.846297:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25661440, count: 4096 00000080:00200000:0.0:1789760766.846304:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25661440, bytes: 4096 00000080:00200000:0.0:1789760766.846314:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.846322:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.846324:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25661440, 25665536) 0 0 00000080:00200000:0.0:1789760766.846329:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.846345:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6265, 6265] 00020000:00200000:0.0:1789760766.846353:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25661440 chunk: [25661440, 25665536) 25665536, 4096 00020000:00200000:0.0:1789760766.846358:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.846376:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6265, 6265] 00000008:00010000:0.0:1789760766.846392:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.846398:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.846410:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25661440, 25665536) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.846432:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25661440=0x1879000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.846488:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.846492:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25665536 00000020:00010000:0.0:1789760766.846501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.846504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.846508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.846512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.846515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.846522:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.846529:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.846544:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760766.846666:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 25661440<>25665536 00000100:00100000:1.0:1789760766.846730:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884160:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.892829:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973582 gen 2 00000008:00010000:1.0:1789760766.892885:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760766.892893:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.892905:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.892930:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760766.892961:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884160:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760766.893113:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6265, 6265] 00000080:00200000:0.0:1789760766.893130:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.893136:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.893153:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25665536 i_blocks 51209 00000020:00200000:0.0:1789760766.893156:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.893165:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.893212:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25665536, count: 4096 00000080:00200000:0.0:1789760766.893220:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25665536, bytes: 4096 00000080:00200000:0.0:1789760766.893281:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.893291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.893294:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25665536, 25669632) 0 0 00000080:00200000:0.0:1789760766.893299:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.893316:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6266, 6266] 00020000:00200000:0.0:1789760766.893324:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25665536 chunk: [25665536, 25669632) 25669632, 4096 00020000:00200000:0.0:1789760766.893329:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.893348:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6266, 6266] 00000008:00010000:0.0:1789760766.893365:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.893371:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760766.893383:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25665536, 25669632) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.893397:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25665536=0x187a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.893536:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.893540:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25669632 00000020:00010000:0.0:1789760766.893549:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.893553:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.893556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.893560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.893563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.893570:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.893577:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.893593:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760766.893716:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 25665536<>25669632 00000100:00100000:1.0:1789760766.893998:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495884288:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760766.952671:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973583 gen 2 00000008:00010000:0.0:1789760766.956194:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760766.956212:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.956225:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.956321:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760766.956355:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495884288:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760766.959970:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6266, 6266] 00000080:00200000:1.0:1789760766.960002:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.960010:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.960030:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25669632 i_blocks 51209 00000020:00200000:1.0:1789760766.960036:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.960054:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.960087:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25669632, count: 4096 00000080:00200000:1.0:1789760766.960096:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25669632, bytes: 4096 00000080:00200000:1.0:1789760766.960111:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.960121:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.960123:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25669632, 25673728) 0 0 00000080:00200000:1.0:1789760766.960128:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.960145:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6267, 6267] 00020000:00200000:1.0:1789760766.960154:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25669632 chunk: [25669632, 25673728) 25673728, 4096 00020000:00200000:1.0:1789760766.960159:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.960199:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6267, 6267] 00000008:00010000:1.0:1789760766.960221:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.960226:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760766.960241:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25669632, 25673728) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.960258:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25669632=0x187b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.960329:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.960332:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25673728 00000020:00010000:1.0:1789760766.960340:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.960344:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.960347:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.960350:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.960353:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.960362:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.960368:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.960396:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760766.961275:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 25669632<>25673728 00000100:00100000:1.0:1789760766.961336:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760766.961365:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495884416:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.037649:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973584 gen 2 00000008:00010000:1.0:1789760767.037699:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760767.037707:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.037717:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.037738:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760767.037767:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495884416:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.038685:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6267, 6267] 00000080:00200000:0.0:1789760767.038706:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.038713:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.038729:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25673728 i_blocks 51209 00000020:00200000:0.0:1789760767.038734:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.038745:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.038778:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25673728, count: 4096 00000080:00200000:0.0:1789760767.038786:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25673728, bytes: 4096 00000080:00200000:0.0:1789760767.038797:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.038806:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.038809:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25673728, 25677824) 0 0 00000080:00200000:0.0:1789760767.038814:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.038832:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6268, 6268] 00020000:00200000:0.0:1789760767.038840:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25673728 chunk: [25673728, 25677824) 25677824, 4096 00020000:00200000:0.0:1789760767.038845:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.038867:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6268, 6268] 00000008:00010000:0.0:1789760767.038885:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.038891:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.038901:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25673728, 25677824) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.038917:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25673728=0x187c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.039755:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.039760:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25677824 00000020:00010000:0.0:1789760767.039771:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.039775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.039779:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.039783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.039786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.039795:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.039803:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.039825:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760767.039958:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 25673728<>25677824 00000100:00100000:1.0:1789760767.040088:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495884544:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.097939:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973585 gen 2 00000008:00010000:1.0:1789760767.098106:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760767.098115:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.098127:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.098250:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760767.098280:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495884544:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.098480:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6268, 6268] 00000080:00200000:0.0:1789760767.098497:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.098503:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.098518:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25677824 i_blocks 51209 00000020:00200000:0.0:1789760767.098523:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.098532:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.098564:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25677824, count: 4096 00000080:00200000:0.0:1789760767.098571:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25677824, bytes: 4096 00000080:00200000:0.0:1789760767.098586:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.098596:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.098598:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25677824, 25681920) 0 0 00000080:00200000:0.0:1789760767.098603:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.098621:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6269, 6269] 00020000:00200000:0.0:1789760767.098629:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25677824 chunk: [25677824, 25681920) 25681920, 4096 00020000:00200000:0.0:1789760767.098634:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.098655:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6269, 6269] 00000008:00010000:0.0:1789760767.098673:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.098679:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.098690:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25677824, 25681920) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.099795:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25677824=0x187d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.101641:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.101651:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25681920 00000020:00010000:0.0:1789760767.101666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.101671:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.101676:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.101680:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.101683:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.101697:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.101707:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.101740:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760767.101885:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 25677824<>25681920 00000100:00100000:0.0:1789760767.101990:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884672:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.167716:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973586 gen 2 00000008:00010000:1.0:1789760767.167766:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760767.167774:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.167785:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.167808:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760767.167834:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884672:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.167935:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6269, 6269] 00000080:00200000:0.0:1789760767.167950:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.167956:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.167972:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25681920 i_blocks 51209 00000020:00200000:0.0:1789760767.167975:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.167984:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.168017:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25681920, count: 4096 00000080:00200000:0.0:1789760767.168024:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25681920, bytes: 4096 00000080:00200000:0.0:1789760767.168032:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.168041:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.168043:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25681920, 25686016) 0 0 00000080:00200000:0.0:1789760767.168048:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.168064:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6270, 6270] 00020000:00200000:0.0:1789760767.168072:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25681920 chunk: [25681920, 25686016) 25686016, 4096 00020000:00200000:0.0:1789760767.168077:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.168097:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6270, 6270] 00000008:00010000:0.0:1789760767.168117:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.168122:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.168133:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25681920, 25686016) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.168148:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25681920=0x187e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.168246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.168250:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25686016 00000020:00010000:0.0:1789760767.168258:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.168262:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.168265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.168269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.168272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.168279:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.168286:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.168302:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760767.168506:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 25681920<>25686016 00000100:00100000:1.0:1789760767.169983:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760767.170030:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884800:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760767.211754:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760767.211772:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.211782:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.212364:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760767.212394:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884800:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.213462:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6270, 6270] 00000080:00200000:0.0:1789760767.213484:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.213490:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.213507:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25686016 i_blocks 51209 00000020:00200000:0.0:1789760767.213511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.213523:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.213559:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25686016, count: 4096 00000080:00200000:0.0:1789760767.213567:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25686016, bytes: 4096 00000080:00200000:0.0:1789760767.213581:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.213591:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.213593:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25686016, 25690112) 0 0 00000080:00200000:0.0:1789760767.213597:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.213613:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6271, 6271] 00020000:00200000:0.0:1789760767.213622:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25686016 chunk: [25686016, 25690112) 25690112, 4096 00020000:00200000:0.0:1789760767.213627:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.213648:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6271, 6271] 00000008:00010000:0.0:1789760767.213667:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.213674:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.213685:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25686016, 25690112) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.213701:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25686016=0x187f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.213776:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.213783:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25690112 00000020:00010000:0.0:1789760767.213803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.213807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.213810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.213813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.213816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.213825:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.213832:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.213855:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760767.213999:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 25686016<>25690112 00000100:00100000:1.0:1789760767.214392:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884928:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.274946:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973587 gen 2 00000100:00100000:1.0:1789760767.274986:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211834700 x1876699495884928/t4294973588(4294973588) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760783 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760767.275047:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760767.275053:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.275060:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.275076:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760767.275098:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495884928:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.275102:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6271, 6271] 00000080:00200000:0.0:1789760767.275119:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.275125:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.275140:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25690112 i_blocks 51209 00000020:00200000:0.0:1789760767.275144:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.275153:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.275215:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25690112, count: 4096 00000080:00200000:0.0:1789760767.275223:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25690112, bytes: 4096 00000080:00200000:0.0:1789760767.275232:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.275240:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.275243:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25690112, 25694208) 0 0 00000080:00200000:0.0:1789760767.275247:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.275265:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6272, 6272] 00020000:00200000:0.0:1789760767.275275:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25690112 chunk: [25690112, 25694208) 25694208, 4096 00020000:00200000:0.0:1789760767.275280:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.275298:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6272, 6272] 00000008:00010000:0.0:1789760767.275316:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.275321:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.275333:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25690112, 25694208) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.275347:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25690112=0x1880000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.275402:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.275406:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25694208 00000020:00010000:0.0:1789760767.275425:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.275428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.275431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.275434:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.275437:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.275445:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.275629:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.275648:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760767.275826:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 25690112<>25694208 00000100:00100000:1.0:1789760767.275964:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495885056:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.323944:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973589 gen 2 00000008:00010000:1.0:1789760767.324014:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760767.324022:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.324033:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.324056:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760767.324087:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495885056:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.324256:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6272, 6272] 00000080:00200000:0.0:1789760767.324272:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.324279:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.324296:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25694208 i_blocks 51209 00000020:00200000:0.0:1789760767.324299:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.324308:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.324354:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25694208, count: 4096 00000080:00200000:0.0:1789760767.324361:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25694208, bytes: 4096 00000080:00200000:0.0:1789760767.324374:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.324382:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.324385:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25694208, 25698304) 0 0 00000080:00200000:0.0:1789760767.324390:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.324408:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6273, 6273] 00020000:00200000:0.0:1789760767.324428:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25694208 chunk: [25694208, 25698304) 25698304, 4096 00020000:00200000:0.0:1789760767.324434:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.324563:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6273, 6273] 00000008:00010000:0.0:1789760767.324582:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.324588:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.324601:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25694208, 25698304) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.324616:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25694208=0x1881000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.324675:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.324678:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25698304 00000020:00010000:0.0:1789760767.324685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.324689:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.324692:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.324695:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.324698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.324705:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.324712:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.324729:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760767.324858:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 25694208<>25698304 00000100:00100000:1.0:1789760767.325053:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495885184:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.381445:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973590 gen 2 00000008:00010000:1.0:1789760767.381500:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760767.381509:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.381521:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.381543:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760767.381574:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495885184:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.381683:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6273, 6273] 00000080:00200000:0.0:1789760767.381701:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.381707:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.381722:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25698304 i_blocks 51209 00000020:00200000:0.0:1789760767.381726:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.381735:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.381775:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25698304, count: 4096 00000080:00200000:0.0:1789760767.381781:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25698304, bytes: 4096 00000080:00200000:0.0:1789760767.381798:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.381807:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.381809:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25698304, 25702400) 0 0 00000080:00200000:0.0:1789760767.381814:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.381831:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6274, 6274] 00020000:00200000:0.0:1789760767.381840:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25698304 chunk: [25698304, 25702400) 25702400, 4096 00020000:00200000:0.0:1789760767.381844:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.381864:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6274, 6274] 00000008:00010000:0.0:1789760767.381882:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.381888:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.381900:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25698304, 25702400) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.381914:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25698304=0x1882000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.381972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.381975:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25702400 00000020:00010000:0.0:1789760767.381983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.381987:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.381990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.381993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.381996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.382003:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.382009:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.382025:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760767.382151:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 25698304<>25702400 00000100:00100000:1.0:1789760767.382375:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495885312:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760767.437967:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760767.442482:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973591 gen 2 00000008:00010000:1.0:1789760767.442534:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760767.442551:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.442563:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.442584:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760767.442614:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495885312:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.442766:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6274, 6274] 00000080:00200000:0.0:1789760767.442783:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.442790:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.442805:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25702400 i_blocks 51209 00000020:00200000:0.0:1789760767.442809:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.442820:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.442856:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25702400, count: 4096 00000080:00200000:0.0:1789760767.442864:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25702400, bytes: 4096 00000080:00200000:0.0:1789760767.442874:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.442883:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.442885:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25702400, 25706496) 0 0 00000080:00200000:0.0:1789760767.442890:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.442910:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6275, 6275] 00020000:00200000:0.0:1789760767.442918:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25702400 chunk: [25702400, 25706496) 25706496, 4096 00020000:00200000:0.0:1789760767.442923:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.442943:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6275, 6275] 00000008:00010000:0.0:1789760767.442963:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.442969:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.442980:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25702400, 25706496) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.442994:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25702400=0x1883000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.443050:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.443054:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25706496 00000020:00010000:0.0:1789760767.443062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.443066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.443069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.443072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.443074:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.443081:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.443089:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.443105:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760767.443286:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25702400<>25706496 00000100:00100000:1.0:1789760767.443820:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495885440:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760767.496600:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760767.496616:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.496628:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.496652:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760767.496681:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495885440:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.496843:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6275, 6275] 00000080:00200000:0.0:1789760767.496861:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.496868:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.496884:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25706496 i_blocks 51209 00000020:00200000:0.0:1789760767.496887:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.496897:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.496931:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25706496, count: 4096 00000080:00200000:0.0:1789760767.496939:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25706496, bytes: 4096 00000080:00200000:0.0:1789760767.496954:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.496963:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.496965:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25706496, 25710592) 0 0 00000080:00200000:0.0:1789760767.496969:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.496986:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6276, 6276] 00020000:00200000:0.0:1789760767.496994:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25706496 chunk: [25706496, 25710592) 25710592, 4096 00020000:00200000:0.0:1789760767.496999:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.497020:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6276, 6276] 00000008:00010000:0.0:1789760767.497039:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.497045:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.497056:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25706496, 25710592) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.497070:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25706496=0x1884000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.497126:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.497129:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25710592 00000020:00010000:0.0:1789760767.497137:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.497140:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.497143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.497147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.497150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.497156:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.497163:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.497217:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760767.497331:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 25706496<>25710592 00000100:00100000:1.0:1789760767.498539:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495885568:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.563610:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973593 gen 2 00000008:00010000:1.0:1789760767.563678:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760767.563686:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.563697:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.563726:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760767.563759:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495885568:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.563975:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6276, 6276] 00000080:00200000:0.0:1789760767.563993:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.563999:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.564015:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25710592 i_blocks 51209 00000020:00200000:0.0:1789760767.564019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.564028:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.564061:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25710592, count: 4096 00000080:00200000:0.0:1789760767.564068:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25710592, bytes: 4096 00000080:00200000:0.0:1789760767.564083:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.564093:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.564095:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25710592, 25714688) 0 0 00000080:00200000:0.0:1789760767.564099:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.564117:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6277, 6277] 00020000:00200000:0.0:1789760767.564125:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25710592 chunk: [25710592, 25714688) 25714688, 4096 00020000:00200000:0.0:1789760767.564130:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.564278:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6277, 6277] 00000008:00010000:0.0:1789760767.564302:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.564309:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.564322:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25710592, 25714688) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.564338:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25710592=0x1885000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.564393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.564396:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25714688 00000020:00010000:0.0:1789760767.564405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.564409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.564521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.564526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.564528:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.564538:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.564545:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.564564:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760767.564701:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 25710592<>25714688 00000100:00100000:1.0:1789760767.565829:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495885696:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.622701:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973594 gen 2 00000008:00010000:1.0:1789760767.622756:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760767.622765:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.622776:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.622795:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760767.622827:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495885696:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.622829:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6277, 6277] 00000080:00200000:0.0:1789760767.622850:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.622856:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.622873:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25714688 i_blocks 51209 00000020:00200000:0.0:1789760767.622876:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.622887:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.622924:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25714688, count: 4096 00000080:00200000:0.0:1789760767.623070:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25714688, bytes: 4096 00000080:00200000:0.0:1789760767.623081:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.623089:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.623091:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25714688, 25718784) 0 0 00000080:00200000:0.0:1789760767.623095:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.623111:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6278, 6278] 00020000:00200000:0.0:1789760767.623118:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25714688 chunk: [25714688, 25718784) 25718784, 4096 00020000:00200000:0.0:1789760767.623123:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.623141:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6278, 6278] 00000008:00010000:0.0:1789760767.623157:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.623162:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.623203:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25714688, 25718784) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.623215:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25714688=0x1886000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.623265:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.623268:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25718784 00000020:00010000:0.0:1789760767.623275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.623279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.623281:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.623285:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.623287:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.623293:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.623299:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.623314:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760767.624156:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 25714688<>25718784 00000100:00100000:1.0:1789760767.624262:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495885824:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760767.659585:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973595 gen 2 00000008:00010000:1.0:1789760767.659638:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760767.659646:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.659658:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.659682:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760767.659713:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495885824:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760767.659919:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6278, 6278] 00000080:00200000:3.0:1789760767.659936:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.659943:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.659965:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25718784 i_blocks 51209 00000020:00200000:3.0:1789760767.659969:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.659978:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.660016:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25718784, count: 4096 00000080:00200000:3.0:1789760767.660025:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25718784, bytes: 4096 00000080:00200000:3.0:1789760767.660035:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.660044:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.660046:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25718784, 25722880) 0 0 00000080:00200000:3.0:1789760767.660051:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.660067:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6279, 6279] 00020000:00200000:3.0:1789760767.660075:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25718784 chunk: [25718784, 25722880) 25722880, 4096 00020000:00200000:3.0:1789760767.660080:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.660101:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6279, 6279] 00000008:00010000:3.0:1789760767.660122:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.660128:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760767.660141:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25718784, 25722880) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.660155:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25718784=0x1887000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.660253:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.660256:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25722880 00000020:00010000:3.0:1789760767.660265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.660269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.660272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.660276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.660279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.660287:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.660294:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.660311:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760767.660459:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 25718784<>25722880 00000100:00100000:2.0:1789760767.660534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495885952:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760767.711886:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973596 gen 2 00000008:00010000:2.0:1789760767.711935:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760767.711944:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760767.711954:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760767.712459:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760767.712491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495885952:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760767.712692:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6279, 6279] 00000080:00200000:3.0:1789760767.712710:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.712717:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.712817:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25722880 i_blocks 51209 00000020:00200000:3.0:1789760767.712822:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.712832:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.712867:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25722880, count: 4096 00000080:00200000:3.0:1789760767.712874:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25722880, bytes: 4096 00000080:00200000:3.0:1789760767.712886:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.712894:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.712896:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25722880, 25726976) 0 0 00000080:00200000:3.0:1789760767.712900:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.712916:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6280, 6280] 00020000:00200000:3.0:1789760767.712925:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25722880 chunk: [25722880, 25726976) 25726976, 4096 00020000:00200000:3.0:1789760767.712929:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.712950:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6280, 6280] 00000008:00010000:3.0:1789760767.714612:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.714626:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760767.714645:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25722880, 25726976) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.714665:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25722880=0x1888000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.714814:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.714817:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25726976 00000020:00010000:3.0:1789760767.714827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.714831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.714834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.714837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.714839:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.714852:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.714861:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.714894:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760767.715540:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 25722880<>25726976 00000100:00100000:2.0:1789760767.716088:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760767.716117:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495886080:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760767.765061:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973597 gen 2 00000008:00010000:2.0:1789760767.765117:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760767.765125:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760767.765137:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760767.765212:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760767.765238:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495886080:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760767.765462:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6280, 6280] 00000080:00200000:3.0:1789760767.765479:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.765485:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.765501:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25726976 i_blocks 51209 00000020:00200000:3.0:1789760767.765505:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.765515:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.765550:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25726976, count: 4096 00000080:00200000:3.0:1789760767.765557:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25726976, bytes: 4096 00000080:00200000:3.0:1789760767.765567:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.765576:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.765578:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25726976, 25731072) 0 0 00000080:00200000:3.0:1789760767.765583:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.765601:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6281, 6281] 00020000:00200000:3.0:1789760767.765611:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25726976 chunk: [25726976, 25731072) 25731072, 4096 00020000:00200000:3.0:1789760767.765616:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.765635:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6281, 6281] 00000008:00010000:3.0:1789760767.765653:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.765660:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760767.765672:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25726976, 25731072) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.765686:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25726976=0x1889000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.765815:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.765818:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25731072 00000020:00010000:3.0:1789760767.765827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.765831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.765834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.765837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.765841:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.765849:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.765855:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.765872:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760767.765999:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 25726976<>25731072 00000100:00100000:2.0:1789760767.766291:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495886208:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760767.815365:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973598 gen 2 00000008:00010000:2.0:1789760767.815428:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760767.815436:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760767.815447:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760767.815470:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:2.0:1789760767.815497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495886208:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760767.815606:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6281, 6281] 00000080:00200000:3.0:1789760767.815623:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.815629:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.815643:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25731072 i_blocks 51209 00000020:00200000:3.0:1789760767.815647:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.815655:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.815687:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25731072, count: 4096 00000080:00200000:3.0:1789760767.815694:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25731072, bytes: 4096 00000080:00200000:3.0:1789760767.815702:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.815710:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.815712:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25731072, 25735168) 0 0 00000080:00200000:3.0:1789760767.815716:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.815731:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6282, 6282] 00020000:00200000:3.0:1789760767.815738:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25731072 chunk: [25731072, 25735168) 25735168, 4096 00020000:00200000:3.0:1789760767.815743:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.815760:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6282, 6282] 00000008:00010000:3.0:1789760767.815776:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.815780:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760767.815791:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25731072, 25735168) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.815803:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25731072=0x188a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.815851:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.815854:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25735168 00000020:00010000:3.0:1789760767.815861:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.815864:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.815867:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.815870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.815873:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.815879:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.815885:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.815899:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760767.816018:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 25731072<>25735168 00000100:00100000:2.0:1789760767.816105:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760767.816122:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495886336:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760767.855163:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973599 gen 2 00000008:00010000:2.0:1789760767.855262:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:2.0:1789760767.855270:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760767.855281:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760767.855302:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:2.0:1789760767.855336:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495886336:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760767.855551:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6282, 6282] 00000080:00200000:3.0:1789760767.855568:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.855575:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.855590:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25735168 i_blocks 51209 00000020:00200000:3.0:1789760767.855593:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.855603:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.855638:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25735168, count: 4096 00000080:00200000:3.0:1789760767.855645:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25735168, bytes: 4096 00000080:00200000:3.0:1789760767.855654:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.855663:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.855666:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25735168, 25739264) 0 0 00000080:00200000:3.0:1789760767.855671:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.855687:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6283, 6283] 00020000:00200000:3.0:1789760767.855696:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25735168 chunk: [25735168, 25739264) 25739264, 4096 00020000:00200000:3.0:1789760767.855701:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.855719:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6283, 6283] 00000008:00010000:3.0:1789760767.855736:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.855755:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760767.855767:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25735168, 25739264) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.855781:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25735168=0x188b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.855842:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.855846:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25739264 00000020:00010000:3.0:1789760767.855865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.855869:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.855872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.855877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.855880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.855888:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.855894:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.856082:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760767.856248:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 25735168<>25739264 00000100:00100000:3.0:1789760767.856299:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760767.856320:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495886464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760767.895504:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973600 gen 2 00000008:00010000:3.0:1789760767.895556:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:3.0:1789760767.895565:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.895576:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.895660:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:3.0:1789760767.895691:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495886464:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760767.895819:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6283, 6283] 00000080:00200000:1.0:1789760767.895835:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760767.895842:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760767.895859:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25739264 i_blocks 51209 00000020:00200000:1.0:1789760767.895863:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760767.895873:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760767.895908:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25739264, count: 4096 00000080:00200000:1.0:1789760767.895914:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25739264, bytes: 4096 00000080:00200000:1.0:1789760767.895924:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760767.895933:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760767.895936:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25739264, 25743360) 0 0 00000080:00200000:1.0:1789760767.895940:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760767.895961:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6284, 6284] 00020000:00200000:1.0:1789760767.895970:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25739264 chunk: [25739264, 25743360) 25743360, 4096 00020000:00200000:1.0:1789760767.895975:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760767.895994:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6284, 6284] 00000008:00010000:1.0:1789760767.896013:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760767.896018:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760767.896030:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25739264, 25743360) append 0 append lockpos 0 00000080:00200000:1.0:1789760767.896043:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25739264=0x188c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760767.896164:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760767.896236:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25743360 00000020:00010000:1.0:1789760767.896247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760767.896251:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.896255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.896257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760767.896260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760767.896269:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760767.896277:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760767.896294:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760767.896789:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 25739264<>25743360 00000100:00100000:0.0:1789760767.896843:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495886592:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760767.926804:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760767.926820:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.926831:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.926896:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760767.926926:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495886592:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760767.926998:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6284, 6284] 00000080:00200000:3.0:1789760767.927014:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.927020:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.927035:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25743360 i_blocks 51209 00000020:00200000:3.0:1789760767.927038:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.927048:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.927165:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25743360, count: 4096 00000080:00200000:3.0:1789760767.927203:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25743360, bytes: 4096 00000080:00200000:3.0:1789760767.927214:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.927223:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.927225:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25743360, 25747456) 0 0 00000080:00200000:3.0:1789760767.927229:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.927245:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6285, 6285] 00020000:00200000:3.0:1789760767.927253:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25743360 chunk: [25743360, 25747456) 25747456, 4096 00020000:00200000:3.0:1789760767.927258:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.927277:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6285, 6285] 00000008:00010000:3.0:1789760767.927293:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.927299:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760767.927311:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25743360, 25747456) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.927325:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25743360=0x188d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.927378:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.927381:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25747456 00000020:00010000:3.0:1789760767.927564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.927570:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.927573:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.927577:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.927581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.927589:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.927596:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.927613:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760767.927802:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 25743360<>25747456 00000100:00100000:2.0:1789760767.927891:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495886720:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760767.958401:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973602 gen 2 00000008:00010000:2.0:1789760767.958855:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760767.958866:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760767.958879:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760767.958902:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760767.958938:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495886720:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760767.959058:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6285, 6285] 00000080:00200000:3.0:1789760767.959076:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.959083:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.959100:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25747456 i_blocks 51209 00000020:00200000:3.0:1789760767.959103:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.959113:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.959149:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25747456, count: 4096 00000080:00200000:3.0:1789760767.959157:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25747456, bytes: 4096 00000080:00200000:3.0:1789760767.959193:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.959202:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.959205:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25747456, 25751552) 0 0 00000080:00200000:3.0:1789760767.959209:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.959227:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6286, 6286] 00020000:00200000:3.0:1789760767.959234:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25747456 chunk: [25747456, 25751552) 25751552, 4096 00020000:00200000:3.0:1789760767.959239:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.959258:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6286, 6286] 00000008:00010000:3.0:1789760767.959276:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.959283:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760767.959296:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25747456, 25751552) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.959311:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25747456=0x188e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.959369:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.959373:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25751552 00000020:00010000:3.0:1789760767.959383:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.959387:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.959390:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.959393:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.959397:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.959405:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.959421:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.959506:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760767.959656:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 25747456<>25751552 00000100:00100000:2.0:1789760767.959727:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495886848:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760767.996999:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973603 gen 2 00000008:00010000:2.0:1789760767.997051:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:2.0:1789760767.997060:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760767.997070:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760767.997091:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:2.0:1789760767.997123:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495886848:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760767.997256:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6286, 6286] 00000080:00200000:0.0:1789760767.997273:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.997279:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.997294:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25751552 i_blocks 51209 00000020:00200000:0.0:1789760767.997297:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.997306:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.997341:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25751552, count: 4096 00000080:00200000:0.0:1789760767.997348:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25751552, bytes: 4096 00000080:00200000:0.0:1789760767.997358:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.997366:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.997369:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25751552, 25755648) 0 0 00000080:00200000:0.0:1789760767.997374:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.997391:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6287, 6287] 00020000:00200000:0.0:1789760767.997400:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25751552 chunk: [25751552, 25755648) 25755648, 4096 00020000:00200000:0.0:1789760767.997405:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.997586:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6287, 6287] 00000008:00010000:0.0:1789760767.997609:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.997616:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760767.997630:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25751552, 25755648) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.997646:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25751552=0x188f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.997717:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.997720:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25755648 00000020:00010000:0.0:1789760767.997796:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.997800:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.997803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.997806:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.997809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.997820:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.997828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.997847:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760767.998068:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 25751552<>25755648 00000100:00100000:1.0:1789760767.998163:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495886976:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.053751:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973604 gen 2 00000008:00010000:1.0:1789760768.053804:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760768.053811:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.053822:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.053851:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760768.053885:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495886976:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.054602:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6287, 6287] 00000080:00200000:0.0:1789760768.054621:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.054629:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.054645:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25755648 i_blocks 51209 00000020:00200000:0.0:1789760768.054649:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.054659:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.054692:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25755648, count: 4096 00000080:00200000:0.0:1789760768.054700:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25755648, bytes: 4096 00000080:00200000:0.0:1789760768.054715:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.054724:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.054727:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25755648, 25759744) 0 0 00000080:00200000:0.0:1789760768.054743:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.054760:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6288, 6288] 00020000:00200000:0.0:1789760768.054768:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25755648 chunk: [25755648, 25759744) 25759744, 4096 00020000:00200000:0.0:1789760768.054773:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.054795:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6288, 6288] 00000008:00010000:0.0:1789760768.054814:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.054819:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.054831:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25755648, 25759744) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.054848:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25755648=0x1890000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.054910:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.054913:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25759744 00000020:00010000:0.0:1789760768.054921:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.054925:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.054929:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.054932:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.054936:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.054943:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.054951:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.054967:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760768.055087:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 25755648<>25759744 00000100:00100000:1.0:1789760768.055283:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495887104:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.116357:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973605 gen 2 00000008:00010000:1.0:1789760768.116416:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760768.116425:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.116448:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.116469:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760768.116497:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495887104:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.116828:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6288, 6288] 00000080:00200000:0.0:1789760768.116846:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.116852:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.116868:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25759744 i_blocks 51209 00000020:00200000:0.0:1789760768.116872:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.116881:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.116918:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25759744, count: 4096 00000080:00200000:0.0:1789760768.116925:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25759744, bytes: 4096 00000080:00200000:0.0:1789760768.116939:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.116948:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.116952:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25759744, 25763840) 0 0 00000080:00200000:0.0:1789760768.116956:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.116973:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6289, 6289] 00020000:00200000:0.0:1789760768.116981:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25759744 chunk: [25759744, 25763840) 25763840, 4096 00020000:00200000:0.0:1789760768.116986:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.117005:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6289, 6289] 00000008:00010000:0.0:1789760768.117024:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.117030:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.117042:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25759744, 25763840) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.117057:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25759744=0x1891000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.117118:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.117121:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25763840 00000020:00010000:0.0:1789760768.117130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.117134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.117137:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.117141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.117144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.117152:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.117159:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.117235:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760768.117357:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 25759744<>25763840 00000100:00100000:1.0:1789760768.118315:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495887232:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760768.163015:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760768.163033:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.163046:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.163062:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760768.163092:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495887232:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.170255:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6289, 6289] 00000080:00200000:0.0:1789760768.170353:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.170362:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.170381:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25763840 i_blocks 51209 00000020:00200000:0.0:1789760768.170386:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.170402:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.170446:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25763840, count: 4096 00000080:00200000:0.0:1789760768.170456:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25763840, bytes: 4096 00000080:00200000:0.0:1789760768.170472:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.170481:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.170484:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25763840, 25767936) 0 0 00000080:00200000:0.0:1789760768.170489:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.170507:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6290, 6290] 00020000:00200000:0.0:1789760768.170515:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25763840 chunk: [25763840, 25767936) 25767936, 4096 00020000:00200000:0.0:1789760768.170519:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.170541:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6290, 6290] 00000008:00010000:0.0:1789760768.170560:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.170567:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.170580:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25763840, 25767936) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.170596:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25763840=0x1892000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.170882:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.170886:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25767936 00000020:00010000:0.0:1789760768.170896:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.170900:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.170904:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.170908:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.170911:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.170920:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.170928:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.171034:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760768.171438:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 25763840<>25767936 00000100:00100000:1.0:1789760768.171542:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495887360:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.214669:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973607 gen 2 00000008:00010000:1.0:1789760768.214734:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760768.214752:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.214763:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.214788:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760768.214820:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495887360:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.215031:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6290, 6290] 00000080:00200000:0.0:1789760768.215048:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.215054:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.215070:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25767936 i_blocks 51209 00000020:00200000:0.0:1789760768.215074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.215084:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.215117:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25767936, count: 4096 00000080:00200000:0.0:1789760768.215124:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25767936, bytes: 4096 00000080:00200000:0.0:1789760768.215137:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.215146:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.215150:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25767936, 25772032) 0 0 00000080:00200000:0.0:1789760768.215154:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.215211:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6291, 6291] 00020000:00200000:0.0:1789760768.215220:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25767936 chunk: [25767936, 25772032) 25772032, 4096 00020000:00200000:0.0:1789760768.215224:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.215244:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6291, 6291] 00000008:00010000:0.0:1789760768.215259:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.215264:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.215275:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25767936, 25772032) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.215288:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25767936=0x1893000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.215358:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.215362:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25772032 00000020:00010000:0.0:1789760768.215370:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.215374:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.215378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.215381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.215384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.215392:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.215400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.215424:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760768.215697:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 25767936<>25772032 00000100:00100000:1.0:1789760768.216058:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495887488:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.261890:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973608 gen 2 00000008:00010000:1.0:1789760768.261949:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760768.261958:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.261969:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.261991:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760768.262390:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495887488:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.262510:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6291, 6291] 00000080:00200000:0.0:1789760768.262527:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.262534:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.262549:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25772032 i_blocks 51209 00000020:00200000:0.0:1789760768.262553:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.262562:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.262594:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25772032, count: 4096 00000080:00200000:0.0:1789760768.262601:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25772032, bytes: 4096 00000080:00200000:0.0:1789760768.262613:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.262622:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.262625:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25772032, 25776128) 0 0 00000080:00200000:0.0:1789760768.262629:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.262647:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6292, 6292] 00020000:00200000:0.0:1789760768.262655:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25772032 chunk: [25772032, 25776128) 25776128, 4096 00020000:00200000:0.0:1789760768.262660:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.262681:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6292, 6292] 00000008:00010000:0.0:1789760768.262699:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.262705:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.262717:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25772032, 25776128) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.262841:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25772032=0x1894000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.262905:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.262908:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25776128 00000020:00010000:0.0:1789760768.262917:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.262920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.262923:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.262927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.262930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.262937:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.262944:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.262959:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760768.263981:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 25772032<>25776128 00000100:00100000:1.0:1789760768.264521:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495887616:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.338463:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973609 gen 2 00000008:00010000:1.0:1789760768.338513:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760768.338522:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.338533:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.338553:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760768.338585:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495887616:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.339542:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6292, 6292] 00000080:00200000:0.0:1789760768.339561:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.339568:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.339585:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25776128 i_blocks 51209 00000020:00200000:0.0:1789760768.339589:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.339599:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.339634:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25776128, count: 4096 00000080:00200000:0.0:1789760768.339642:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25776128, bytes: 4096 00000080:00200000:0.0:1789760768.339653:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.339662:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.339665:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25776128, 25780224) 0 0 00000080:00200000:0.0:1789760768.339670:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.339688:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6293, 6293] 00020000:00200000:0.0:1789760768.339697:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25776128 chunk: [25776128, 25780224) 25780224, 4096 00020000:00200000:0.0:1789760768.339702:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.339720:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6293, 6293] 00000008:00010000:0.0:1789760768.339746:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.339752:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.339762:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25776128, 25780224) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.339775:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25776128=0x1895000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.339834:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.339837:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25780224 00000020:00010000:0.0:1789760768.339844:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.339848:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.339851:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.339854:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.339857:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.339863:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.339869:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.339882:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760768.340143:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 25776128<>25780224 00000100:00100000:1.0:1789760768.340289:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495887744:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.386202:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973610 gen 2 00000008:00010000:1.0:1789760768.386255:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760768.386263:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.386275:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.386297:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760768.386326:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495887744:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.386492:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6293, 6293] 00000080:00200000:0.0:1789760768.386508:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.386514:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.386530:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25780224 i_blocks 51209 00000020:00200000:0.0:1789760768.386534:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.386544:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.386577:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25780224, count: 4096 00000080:00200000:0.0:1789760768.386583:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25780224, bytes: 4096 00000080:00200000:0.0:1789760768.386593:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.386601:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.386604:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25780224, 25784320) 0 0 00000080:00200000:0.0:1789760768.386608:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.386624:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6294, 6294] 00020000:00200000:0.0:1789760768.386631:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25780224 chunk: [25780224, 25784320) 25784320, 4096 00020000:00200000:0.0:1789760768.386636:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.386654:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6294, 6294] 00000008:00010000:0.0:1789760768.386671:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.386677:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.386701:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25780224, 25784320) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.386714:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25780224=0x1896000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.386890:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.386894:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25784320 00000020:00010000:0.0:1789760768.386902:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.386906:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.386909:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.386912:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.386915:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.386921:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.386927:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.386941:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760768.387057:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 25780224<>25784320 00000100:00100000:1.0:1789760768.387232:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495887872:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.417901:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973611 gen 2 00000008:00010000:1.0:1789760768.417945:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760768.417953:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.417963:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.417982:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760768.418008:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495887872:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.418139:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6294, 6294] 00000080:00200000:0.0:1789760768.418155:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.418161:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.418196:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25784320 i_blocks 51209 00000020:00200000:0.0:1789760768.418200:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.418208:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.418239:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25784320, count: 4096 00000080:00200000:0.0:1789760768.418246:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25784320, bytes: 4096 00000080:00200000:0.0:1789760768.418257:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.418266:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.418269:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25784320, 25788416) 0 0 00000080:00200000:0.0:1789760768.418273:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.418290:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6295, 6295] 00020000:00200000:0.0:1789760768.418297:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25784320 chunk: [25784320, 25788416) 25788416, 4096 00020000:00200000:0.0:1789760768.418302:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.418320:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6295, 6295] 00000008:00010000:0.0:1789760768.418337:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.418342:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.418353:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25784320, 25788416) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.418367:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25784320=0x1897000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.418443:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.418446:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25788416 00000020:00010000:0.0:1789760768.418454:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.418457:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.418460:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.418464:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.418467:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.418473:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.418480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.418495:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760768.418620:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 25784320<>25788416 00000100:00100000:1.0:1789760768.419213:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760768.419235:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495888000:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760768.443002:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760768.443017:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.443029:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.443050:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760768.443083:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495888000:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.443320:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6295, 6295] 00000080:00200000:0.0:1789760768.443336:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.443343:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.443360:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25788416 i_blocks 51209 00000020:00200000:0.0:1789760768.443364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.443373:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.443486:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25788416, count: 4096 00000080:00200000:0.0:1789760768.443495:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25788416, bytes: 4096 00000080:00200000:0.0:1789760768.443507:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.443517:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.443520:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25788416, 25792512) 0 0 00000080:00200000:0.0:1789760768.443524:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.443541:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6296, 6296] 00020000:00200000:0.0:1789760768.443550:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25788416 chunk: [25788416, 25792512) 25792512, 4096 00020000:00200000:0.0:1789760768.443554:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.443575:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6296, 6296] 00000008:00010000:0.0:1789760768.443593:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.443599:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.443612:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25788416, 25792512) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.443626:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25788416=0x1898000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.443689:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.443693:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25792512 00000020:00010000:0.0:1789760768.443701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.443706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.443709:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.443712:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.443715:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.443723:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.443744:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.443762:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760768.444015:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 25788416<>25792512 00000100:00100000:1.0:1789760768.444143:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760768.444164:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495888128:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760768.459588:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760768.495023:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973613 gen 2 00000008:00010000:1.0:1789760768.495095:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760768.495103:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.495113:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.495376:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760768.495416:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495888128:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.495613:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6296, 6296] 00000080:00200000:0.0:1789760768.495630:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.495636:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.495652:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25792512 i_blocks 51209 00000020:00200000:0.0:1789760768.495655:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.495665:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.495701:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25792512, count: 4096 00000080:00200000:0.0:1789760768.495707:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25792512, bytes: 4096 00000080:00200000:0.0:1789760768.495722:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.495731:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.495733:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25792512, 25796608) 0 0 00000080:00200000:0.0:1789760768.495738:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.495755:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6297, 6297] 00020000:00200000:0.0:1789760768.495857:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25792512 chunk: [25792512, 25796608) 25796608, 4096 00020000:00200000:0.0:1789760768.495863:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.495885:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6297, 6297] 00000008:00010000:0.0:1789760768.495903:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.495908:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.495920:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25792512, 25796608) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.495936:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25792512=0x1899000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.496003:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.496006:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25796608 00000020:00010000:0.0:1789760768.496016:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.496020:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.496023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.496027:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.496030:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.496038:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.496046:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.496066:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760768.496290:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 25792512<>25796608 00000100:00100000:1.0:1789760768.496375:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495888256:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760768.538898:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760768.538916:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.538930:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.539056:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760768.539090:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495888256:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.539638:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6297, 6297] 00000080:00200000:0.0:1789760768.539659:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.539665:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.539681:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25796608 i_blocks 51209 00000020:00200000:0.0:1789760768.539685:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.539698:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.539732:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25796608, count: 4096 00000080:00200000:0.0:1789760768.539739:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25796608, bytes: 4096 00000080:00200000:0.0:1789760768.539754:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.539763:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.539765:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25796608, 25800704) 0 0 00000080:00200000:0.0:1789760768.539769:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.539786:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6298, 6298] 00020000:00200000:0.0:1789760768.539794:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25796608 chunk: [25796608, 25800704) 25800704, 4096 00020000:00200000:0.0:1789760768.539799:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.539820:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6298, 6298] 00000008:00010000:0.0:1789760768.539836:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.539842:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.539854:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25796608, 25800704) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.539867:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25796608=0x189a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.539931:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.539934:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25800704 00000020:00010000:0.0:1789760768.539942:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.539945:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.539949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.539952:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.539955:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.539962:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.539969:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.539989:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760768.540335:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 25796608<>25800704 00000100:00100000:1.0:1789760768.540440:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495888384:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.609866:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973615 gen 2 00000008:00010000:1.0:1789760768.609936:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760768.609945:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.609955:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.609978:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760768.610010:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495888384:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.610203:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6298, 6298] 00000080:00200000:0.0:1789760768.610220:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.610227:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.610243:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25800704 i_blocks 51209 00000020:00200000:0.0:1789760768.610247:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.610258:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.610296:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25800704, count: 4096 00000080:00200000:0.0:1789760768.610303:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25800704, bytes: 4096 00000080:00200000:0.0:1789760768.610316:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.610327:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.610329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25800704, 25804800) 0 0 00000080:00200000:0.0:1789760768.610334:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.610350:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6299, 6299] 00020000:00200000:0.0:1789760768.610358:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25800704 chunk: [25800704, 25804800) 25804800, 4096 00020000:00200000:0.0:1789760768.610362:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.610381:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6299, 6299] 00000008:00010000:0.0:1789760768.610398:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.610414:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.610425:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25800704, 25804800) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.610441:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25800704=0x189b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.610510:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.610513:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25804800 00000020:00010000:0.0:1789760768.610520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.610524:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.610527:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.610531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.610535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.610542:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.610548:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.611699:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760768.611932:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 25800704<>25804800 00000100:00100000:1.0:1789760768.612158:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495888512:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760768.674263:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760768.674281:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.674294:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.674313:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760768.674343:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495888512:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.675081:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6299, 6299] 00000080:00200000:0.0:1789760768.675103:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.675110:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.675126:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25804800 i_blocks 51209 00000020:00200000:0.0:1789760768.675131:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.675142:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.675355:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25804800, count: 4096 00000080:00200000:0.0:1789760768.675443:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25804800, bytes: 4096 00000080:00200000:0.0:1789760768.675457:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.675467:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.675470:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25804800, 25808896) 0 0 00000080:00200000:0.0:1789760768.675475:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.675492:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6300, 6300] 00020000:00200000:0.0:1789760768.675500:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25804800 chunk: [25804800, 25808896) 25808896, 4096 00020000:00200000:0.0:1789760768.675505:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.675527:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6300, 6300] 00000008:00010000:0.0:1789760768.675547:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.675554:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.675565:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25804800, 25808896) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.675579:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25804800=0x189c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.675629:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.675839:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:1.0:1789760768.675965:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25804800<>25808896 00000100:00100000:1.0:1789760768.676018:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495888640:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.676253:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25808896 00000020:00010000:0.0:1789760768.676264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.676267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.676271:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.676274:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.676277:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.676286:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.676294:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760768.720796:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973617 gen 2 00000008:00010000:1.0:1789760768.720861:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760768.720870:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.720879:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.720942:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760768.720981:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495888640:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760768.721660:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6300, 6300] 00000080:00200000:0.0:1789760768.721745:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.721752:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.721768:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25808896 i_blocks 51209 00000020:00200000:0.0:1789760768.721772:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.721783:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.721815:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25808896, count: 4096 00000080:00200000:0.0:1789760768.721822:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25808896, bytes: 4096 00000080:00200000:0.0:1789760768.721835:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.721843:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.721846:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25808896, 25812992) 0 0 00000080:00200000:0.0:1789760768.721851:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.721868:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6301, 6301] 00020000:00200000:0.0:1789760768.721876:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25808896 chunk: [25808896, 25812992) 25812992, 4096 00020000:00200000:0.0:1789760768.721978:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.722000:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6301, 6301] 00000008:00010000:0.0:1789760768.722018:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.722024:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760768.722036:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25808896, 25812992) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.722052:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25808896=0x189d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.722118:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.722121:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25812992 00000020:00010000:0.0:1789760768.722131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.722134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.722137:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.722141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.722145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.722152:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.722160:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.723022:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760768.723160:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 25808896<>25812992 00000100:00100000:1.0:1789760768.723366:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495888768:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.799426:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973618 gen 2 00000008:00010000:1.0:1789760768.799485:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760768.799493:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.799504:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.799535:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760768.799565:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495888768:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760768.799757:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6301, 6301] 00000080:00200000:2.0:1789760768.799774:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.799781:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.799795:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25812992 i_blocks 51209 00000020:00200000:2.0:1789760768.799799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.799809:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.799841:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25812992, count: 4096 00000080:00200000:2.0:1789760768.799848:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25812992, bytes: 4096 00000080:00200000:2.0:1789760768.799857:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.799867:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.799869:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25812992, 25817088) 0 0 00000080:00200000:2.0:1789760768.799874:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.799889:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6302, 6302] 00020000:00200000:2.0:1789760768.799897:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25812992 chunk: [25812992, 25817088) 25817088, 4096 00020000:00200000:2.0:1789760768.799902:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.799920:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6302, 6302] 00000008:00010000:2.0:1789760768.799938:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.799943:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760768.799955:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25812992, 25817088) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.799967:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25812992=0x189e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.800023:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.800026:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25817088 00000020:00010000:2.0:1789760768.800033:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.800036:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.800039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.800043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.800046:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.800054:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.800060:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.800077:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760768.800247:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 25812992<>25817088 00000100:00100000:3.0:1789760768.800313:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495888896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760768.904813:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973619 gen 2 00000008:00010000:3.0:1789760768.904865:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760768.906737:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.906767:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.906801:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760768.906841:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495888896:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760768.906988:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6302, 6302] 00000080:00200000:2.0:1789760768.907009:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.907015:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.907034:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25817088 i_blocks 51209 00000020:00200000:2.0:1789760768.907039:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.907049:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.907086:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25817088, count: 4096 00000080:00200000:2.0:1789760768.907095:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25817088, bytes: 4096 00000080:00200000:2.0:1789760768.907110:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.907120:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.907122:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25817088, 25821184) 0 0 00000080:00200000:2.0:1789760768.907127:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.907144:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6303, 6303] 00020000:00200000:2.0:1789760768.907152:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25817088 chunk: [25817088, 25821184) 25821184, 4096 00020000:00200000:2.0:1789760768.907157:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.907232:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6303, 6303] 00000008:00010000:2.0:1789760768.907251:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.907257:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760768.907270:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25817088, 25821184) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.907287:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25817088=0x189f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.907340:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.907343:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25821184 00000020:00010000:2.0:1789760768.907351:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.907355:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.907358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.907362:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.907364:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.907372:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.907379:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760768.907684:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495889024:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760768.907822:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495889280:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760768.907859:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760768.907864:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495889152:192.168.203.109@tcp:400:kworker.0 00000008:00010000:3.0:1789760768.907991:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:3.0:1789760768.908111:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 25817088<>25821184 00000100:00100000:3.0:1789760768.908141:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760768.908151:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495889408:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760768.911617:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495889152:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760768.911639:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495889280:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760768.918882:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495889024:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760768.964478:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973620 gen 2 00000008:00010000:3.0:1789760768.964527:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760768.964535:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.964545:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.964574:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760768.964602:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495889408:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760768.964756:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6303, 6303] 00000080:00200000:2.0:1789760768.964772:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.964778:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.964794:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25821184 i_blocks 51209 00000020:00200000:2.0:1789760768.964798:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.964807:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.964844:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25821184, count: 4096 00000080:00200000:2.0:1789760768.964851:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25821184, bytes: 4096 00000080:00200000:2.0:1789760768.964861:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.964869:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.964872:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25821184, 25825280) 0 0 00000080:00200000:2.0:1789760768.964876:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.964892:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6304, 6304] 00020000:00200000:2.0:1789760768.964900:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25821184 chunk: [25821184, 25825280) 25825280, 4096 00020000:00200000:2.0:1789760768.964981:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.965002:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6304, 6304] 00000008:00010000:2.0:1789760768.965022:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.965027:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760768.965040:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25821184, 25825280) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.965054:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25821184=0x18a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.965114:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.965117:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25825280 00000020:00010000:2.0:1789760768.965126:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.965130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.965133:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.965136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.965139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.965146:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.965153:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.965197:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760768.965325:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 25821184<>25825280 00000100:00100000:2.0:1789760768.965372:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760768.965391:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495889536:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760769.024075:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973621 gen 2 00000008:00010000:2.0:1789760769.024131:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760769.024140:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760769.024152:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.024252:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6304, 6304] 00000008:00010000:2.0:1789760769.024256:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000080:00200000:1.0:1789760769.024273:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.024281:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:2.0:1789760769.024286:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495889536:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760769.024300:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25825280 i_blocks 51209 00000020:00200000:1.0:1789760769.024304:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.024315:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.024354:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25825280, count: 4096 00000080:00200000:1.0:1789760769.024363:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25825280, bytes: 4096 00000080:00200000:1.0:1789760769.024373:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.024447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.024450:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25825280, 25829376) 0 0 00000080:00200000:1.0:1789760769.024456:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.024474:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6305, 6305] 00020000:00200000:1.0:1789760769.024482:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25825280 chunk: [25825280, 25829376) 25829376, 4096 00020000:00200000:1.0:1789760769.024487:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.024508:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6305, 6305] 00000008:00010000:1.0:1789760769.024523:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.024527:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760769.024537:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25825280, 25829376) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.024552:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25825280=0x18a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.024598:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.024600:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25829376 00000020:00010000:1.0:1789760769.024607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.024610:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.024612:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.024615:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.024617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.024622:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.024628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.024640:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760769.024884:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25825280<>25829376 00000100:00100000:0.0:1789760769.025029:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495889664:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760769.071480:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973622 gen 2 00000008:00010000:1.0:1789760769.071536:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760769.071545:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.071555:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.071578:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760769.071608:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495889664:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760769.071693:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6305, 6305] 00000080:00200000:0.0:1789760769.071711:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.071718:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.071744:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25829376 i_blocks 51209 00000020:00200000:0.0:1789760769.071749:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.071758:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.071795:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25829376, count: 4096 00000080:00200000:0.0:1789760769.071803:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25829376, bytes: 4096 00000080:00200000:0.0:1789760769.071814:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.071823:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.071826:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25829376, 25833472) 0 0 00000080:00200000:0.0:1789760769.071830:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.071848:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6306, 6306] 00020000:00200000:0.0:1789760769.071856:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25829376 chunk: [25829376, 25833472) 25833472, 4096 00020000:00200000:0.0:1789760769.071861:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.071886:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6306, 6306] 00000008:00010000:0.0:1789760769.072030:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.072037:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760769.072052:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25829376, 25833472) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.072068:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25829376=0x18a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.072127:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.072130:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25833472 00000020:00010000:0.0:1789760769.072139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.072142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.072146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.072149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.072152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.072160:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.072166:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.072211:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760769.072342:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 25829376<>25833472 00000100:00100000:1.0:1789760769.072443:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495889792:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760769.126983:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973623 gen 2 00000008:00010000:1.0:1789760769.127038:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760769.127048:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.127060:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.127084:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760769.127115:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495889792:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760769.127256:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6306, 6306] 00000080:00200000:0.0:1789760769.127274:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.127281:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.127298:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25833472 i_blocks 51209 00000020:00200000:0.0:1789760769.127302:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.127311:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.127348:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25833472, count: 4096 00000080:00200000:0.0:1789760769.127356:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25833472, bytes: 4096 00000080:00200000:0.0:1789760769.127367:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.127385:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.127388:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25833472, 25837568) 0 0 00000080:00200000:0.0:1789760769.127394:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.127420:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6307, 6307] 00020000:00200000:0.0:1789760769.127429:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25833472 chunk: [25833472, 25837568) 25837568, 4096 00020000:00200000:0.0:1789760769.127434:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.127455:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6307, 6307] 00000008:00010000:0.0:1789760769.127474:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.127480:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760769.127492:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25833472, 25837568) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.127506:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25833472=0x18a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.127566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.127569:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25837568 00000020:00010000:0.0:1789760769.127579:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.127583:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.127585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.127589:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.127592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.127600:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.127608:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.127625:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760769.127869:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 25833472<>25837568 00000100:00100000:1.0:1789760769.128039:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760769.128057:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495889920:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760769.167602:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760769.167621:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.167634:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.167656:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000080:00200000:0.0:1789760769.167686:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6307, 6307] 00000100:00100000:1.0:1789760769.167687:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495889920:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760769.167705:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.167711:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.167726:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25837568 i_blocks 51209 00000020:00200000:0.0:1789760769.167730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.167739:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.167986:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25837568, count: 4096 00000080:00200000:0.0:1789760769.167996:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25837568, bytes: 4096 00000080:00200000:0.0:1789760769.168006:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.168015:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.168017:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25837568, 25841664) 0 0 00000080:00200000:0.0:1789760769.168023:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.168112:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6308, 6308] 00020000:00200000:0.0:1789760769.168123:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25837568 chunk: [25837568, 25841664) 25841664, 4096 00020000:00200000:0.0:1789760769.168128:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.168149:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6308, 6308] 00000008:00010000:0.0:1789760769.168247:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.168254:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760769.168268:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25837568, 25841664) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.168284:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25837568=0x18a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.168349:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.168352:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25841664 00000020:00010000:0.0:1789760769.168361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.168365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.168368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.168372:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.168375:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.168461:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.168469:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.168488:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760769.168624:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 25837568<>25841664 00000100:00100000:1.0:1789760769.168704:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495890048:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760769.207522:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973625 gen 2 00000008:00010000:1.0:1789760769.207594:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760769.207604:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.207614:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.207634:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760769.207665:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495890048:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760769.207890:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6308, 6308] 00000080:00200000:0.0:1789760769.207908:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.207915:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.207931:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25841664 i_blocks 51209 00000020:00200000:0.0:1789760769.207934:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.207944:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.207979:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25841664, count: 4096 00000080:00200000:0.0:1789760769.207987:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25841664, bytes: 4096 00000080:00200000:0.0:1789760769.207996:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.208006:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.208009:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25841664, 25845760) 0 0 00000080:00200000:0.0:1789760769.208014:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.208032:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6309, 6309] 00020000:00200000:0.0:1789760769.208685:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25841664 chunk: [25841664, 25845760) 25845760, 4096 00020000:00200000:0.0:1789760769.208694:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.208719:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6309, 6309] 00000008:00010000:0.0:1789760769.208737:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.208744:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760769.208755:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25841664, 25845760) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.208771:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25841664=0x18a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.209166:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.209202:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25845760 00000020:00010000:0.0:1789760769.209212:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.209216:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.209221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.209225:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.209228:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.209238:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.209246:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.209272:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760769.209769:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 25841664<>25845760 00000100:00100000:1.0:1789760769.209931:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760769.210311:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495890176:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760769.256109:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973626 gen 2 00000008:00010000:1.0:1789760769.256157:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760769.256166:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.256215:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.256243:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760769.256273:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495890176:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.256393:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6309, 6309] 00000080:00200000:2.0:1789760769.256419:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.256425:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.256439:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25845760 i_blocks 51209 00000020:00200000:2.0:1789760769.256443:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.256452:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.256490:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25845760, count: 4096 00000080:00200000:2.0:1789760769.256498:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25845760, bytes: 4096 00000080:00200000:2.0:1789760769.256507:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.256515:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.256518:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25845760, 25849856) 0 0 00000080:00200000:2.0:1789760769.256523:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.256540:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6310, 6310] 00020000:00200000:2.0:1789760769.256548:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25845760 chunk: [25845760, 25849856) 25849856, 4096 00020000:00200000:2.0:1789760769.256553:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.256572:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6310, 6310] 00000008:00010000:2.0:1789760769.256589:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.256595:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.256606:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25845760, 25849856) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.256620:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25845760=0x18a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.256674:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.256677:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25849856 00000020:00010000:2.0:1789760769.256686:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.256695:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.256698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.256701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.256704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.256711:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.256718:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.256734:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760769.256891:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 25845760<>25849856 00000100:00100000:2.0:1789760769.256939:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495890304:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760769.285144:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973627 gen 2 00000008:00010000:2.0:1789760769.285235:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760769.285243:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760769.285259:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760769.285288:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760769.285317:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495890304:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760769.285438:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6310, 6310] 00000080:00200000:1.0:1789760769.285455:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.285461:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.285476:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25849856 i_blocks 51209 00000020:00200000:1.0:1789760769.285480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.285490:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.285527:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25849856, count: 4096 00000080:00200000:1.0:1789760769.285534:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25849856, bytes: 4096 00000080:00200000:1.0:1789760769.285549:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.285558:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.285560:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25849856, 25853952) 0 0 00000080:00200000:1.0:1789760769.285568:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.285583:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6311, 6311] 00020000:00200000:1.0:1789760769.285603:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25849856 chunk: [25849856, 25853952) 25853952, 4096 00020000:00200000:1.0:1789760769.285609:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.285629:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6311, 6311] 00000008:00010000:1.0:1789760769.285648:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.285654:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760769.285665:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25849856, 25853952) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.285681:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25849856=0x18a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.285741:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.285744:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25853952 00000020:00010000:1.0:1789760769.285752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.285755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.285758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.285761:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.285764:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.285771:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.285778:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.285794:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760769.285923:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25849856<>25853952 00000100:00100000:0.0:1789760769.285991:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760769.286010:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495890432:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760769.318498:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973628 gen 2 00000008:00010000:1.0:1789760769.319049:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760769.319060:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.319073:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.319103:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760769.319140:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495890432:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.319314:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6311, 6311] 00000080:00200000:2.0:1789760769.319332:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.319339:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.319355:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25853952 i_blocks 51209 00000020:00200000:2.0:1789760769.319359:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.319369:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.319418:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25853952, count: 4096 00000080:00200000:2.0:1789760769.319425:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25853952, bytes: 4096 00000080:00200000:2.0:1789760769.319439:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.319447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.319449:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25853952, 25858048) 0 0 00000080:00200000:2.0:1789760769.319454:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.319470:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6312, 6312] 00020000:00200000:2.0:1789760769.319480:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25853952 chunk: [25853952, 25858048) 25858048, 4096 00020000:00200000:2.0:1789760769.319485:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.319506:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6312, 6312] 00000008:00010000:2.0:1789760769.319524:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.319530:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.319543:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25853952, 25858048) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.319572:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25853952=0x18a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.319635:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.319638:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25858048 00000020:00010000:2.0:1789760769.319646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.319650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.319653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.319657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.319660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.319667:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.319674:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.319690:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760769.319824:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 25853952<>25858048 00000100:00100000:2.0:1789760769.319874:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760769.319892:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495890560:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760769.366226:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760769.366241:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760769.366252:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760769.366352:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760769.366384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495890560:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760769.371251:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6312, 6312] 00000080:00200000:1.0:1789760769.371282:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.371288:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.371310:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25858048 i_blocks 51209 00000020:00200000:1.0:1789760769.371315:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.371334:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.371371:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25858048, count: 4096 00000080:00200000:1.0:1789760769.371380:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25858048, bytes: 4096 00000080:00200000:1.0:1789760769.371396:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.371418:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.371421:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25858048, 25862144) 0 0 00000080:00200000:1.0:1789760769.371425:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.371443:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6313, 6313] 00020000:00200000:1.0:1789760769.371451:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25858048 chunk: [25858048, 25862144) 25862144, 4096 00020000:00200000:1.0:1789760769.371456:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.371478:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6313, 6313] 00000008:00010000:1.0:1789760769.371512:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.371518:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760769.371531:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25858048, 25862144) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.371547:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25858048=0x18a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.371627:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.371630:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25862144 00000020:00010000:1.0:1789760769.371639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.371644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.371647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.371651:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.371654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.371663:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.371670:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.371708:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760769.371980:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25858048<>25862144 00000100:00100000:0.0:1789760769.372084:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495890688:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760769.404157:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973630 gen 2 00000008:00010000:1.0:1789760769.404309:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760769.404318:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.404329:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.404389:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760769.404432:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495890688:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.404471:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6313, 6313] 00000080:00200000:2.0:1789760769.404488:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.404496:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.404513:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25862144 i_blocks 51209 00000020:00200000:2.0:1789760769.404519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.404530:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.404573:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25862144, count: 4096 00000080:00200000:2.0:1789760769.404581:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25862144, bytes: 4096 00000080:00200000:2.0:1789760769.404594:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.404604:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.404607:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25862144, 25866240) 0 0 00000080:00200000:2.0:1789760769.404611:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.404642:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6314, 6314] 00020000:00200000:2.0:1789760769.404653:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25862144 chunk: [25862144, 25866240) 25866240, 4096 00020000:00200000:2.0:1789760769.404658:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.404682:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6314, 6314] 00000008:00010000:2.0:1789760769.404703:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.404711:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.405464:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25862144, 25866240) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.405486:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25862144=0x18aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.405561:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.405565:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25866240 00000020:00010000:2.0:1789760769.405574:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.405578:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.405582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.405585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.405588:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.405599:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.405607:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.405643:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760769.406064:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 25862144<>25866240 00000100:00100000:3.0:1789760769.406248:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495890816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760769.463681:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973631 gen 2 00000008:00010000:3.0:1789760769.463743:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760769.463751:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.463763:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.463788:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760769.463818:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495890816:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.464071:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6314, 6314] 00000080:00200000:2.0:1789760769.464087:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.464093:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.464108:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25866240 i_blocks 51209 00000020:00200000:2.0:1789760769.464112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.464121:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.464154:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25866240, count: 4096 00000080:00200000:2.0:1789760769.464160:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25866240, bytes: 4096 00000080:00200000:2.0:1789760769.464203:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.464213:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.464215:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25866240, 25870336) 0 0 00000080:00200000:2.0:1789760769.464219:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.464236:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6315, 6315] 00020000:00200000:2.0:1789760769.464244:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25866240 chunk: [25866240, 25870336) 25870336, 4096 00020000:00200000:2.0:1789760769.464248:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.464269:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6315, 6315] 00000008:00010000:2.0:1789760769.464288:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.464294:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.464306:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25866240, 25870336) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.464321:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25866240=0x18ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.464388:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.464391:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25870336 00000020:00010000:2.0:1789760769.464409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.464413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.464417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.464421:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.464424:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.464431:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.464438:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.464454:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760769.464698:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 25866240<>25870336 00000100:00100000:3.0:1789760769.465701:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760769.465737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495890944:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760769.483546:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760769.519071:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973632 gen 2 00000008:00010000:3.0:1789760769.519109:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760769.519115:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.519124:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.519143:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760769.519198:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495890944:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.519262:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6315, 6315] 00000080:00200000:2.0:1789760769.519287:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.519294:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.519312:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25870336 i_blocks 51209 00000020:00200000:2.0:1789760769.519317:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.519331:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.519364:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25870336, count: 4096 00000080:00200000:2.0:1789760769.519372:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25870336, bytes: 4096 00000080:00200000:2.0:1789760769.519384:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.519393:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.519395:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25870336, 25874432) 0 0 00000080:00200000:2.0:1789760769.519412:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.519432:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6316, 6316] 00020000:00200000:2.0:1789760769.519682:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25870336 chunk: [25870336, 25874432) 25874432, 4096 00020000:00200000:2.0:1789760769.519688:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.519711:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6316, 6316] 00000008:00010000:2.0:1789760769.519729:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.519735:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.519748:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25870336, 25874432) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.519763:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25870336=0x18ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.519828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.519832:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25874432 00000020:00010000:2.0:1789760769.519840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.519844:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.519847:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.519850:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.519853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.519861:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.519867:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.519888:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760769.520023:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 25870336<>25874432 00000100:00100000:3.0:1789760769.520114:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495891072:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760769.574007:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973633 gen 2 00000008:00010000:3.0:1789760769.574052:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760769.574059:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.574068:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.574092:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760769.574116:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495891072:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760769.574285:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6316, 6316] 00000080:00200000:1.0:1789760769.574299:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.574304:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.574327:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25874432 i_blocks 51209 00000020:00200000:1.0:1789760769.574331:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.574339:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.574366:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25874432, count: 4096 00000080:00200000:1.0:1789760769.574373:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25874432, bytes: 4096 00000080:00200000:1.0:1789760769.574388:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.574395:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.574407:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25874432, 25878528) 0 0 00000080:00200000:1.0:1789760769.574411:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.574423:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6317, 6317] 00020000:00200000:1.0:1789760769.574431:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25874432 chunk: [25874432, 25878528) 25878528, 4096 00020000:00200000:1.0:1789760769.574435:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.574451:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6317, 6317] 00000008:00010000:1.0:1789760769.574464:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.574469:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760769.574477:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25874432, 25878528) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.574499:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25874432=0x18ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.574550:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.574552:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25878528 00000020:00010000:1.0:1789760769.574559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.574562:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.574565:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.574568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.574571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.574577:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.574583:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.574596:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760769.574686:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25874432<>25878528 00000100:00100000:1.0:1789760769.574730:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760769.574743:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495891200:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760769.645320:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973634 gen 2 00000008:00010000:1.0:1789760769.645357:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760769.645364:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.645372:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.645391:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760769.645424:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495891200:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.645438:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6317, 6317] 00000080:00200000:2.0:1789760769.645454:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.645460:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.645472:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25878528 i_blocks 51209 00000020:00200000:2.0:1789760769.645477:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.645552:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.645581:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25878528, count: 4096 00000080:00200000:2.0:1789760769.645587:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25878528, bytes: 4096 00000080:00200000:2.0:1789760769.645596:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.645603:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.645605:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25878528, 25882624) 0 0 00000080:00200000:2.0:1789760769.645609:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.645620:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6318, 6318] 00020000:00200000:2.0:1789760769.645626:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25878528 chunk: [25878528, 25882624) 25882624, 4096 00020000:00200000:2.0:1789760769.645630:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.645646:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6318, 6318] 00000008:00010000:2.0:1789760769.645661:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.645667:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.645676:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25878528, 25882624) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.645687:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25878528=0x18ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.645805:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.645809:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25882624 00000020:00010000:2.0:1789760769.645816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.645819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.645822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.645825:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.645828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.645834:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.645839:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.645914:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760769.646314:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 25878528<>25882624 00000100:00100000:3.0:1789760769.646382:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495891328:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760769.711730:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760769.711742:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.711751:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.711772:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760769.711797:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495891328:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.712035:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6318, 6318] 00000080:00200000:2.0:1789760769.712049:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.712055:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.712069:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25882624 i_blocks 51209 00000020:00200000:2.0:1789760769.712074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.712081:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.712109:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25882624, count: 4096 00000080:00200000:2.0:1789760769.712115:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25882624, bytes: 4096 00000080:00200000:2.0:1789760769.712122:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.712128:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.712131:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25882624, 25886720) 0 0 00000080:00200000:2.0:1789760769.712134:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.712145:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6319, 6319] 00020000:00200000:2.0:1789760769.712152:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25882624 chunk: [25882624, 25886720) 25886720, 4096 00020000:00200000:2.0:1789760769.712155:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.712198:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6319, 6319] 00000008:00010000:2.0:1789760769.712213:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.712218:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.712227:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25882624, 25886720) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.712238:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25882624=0x18af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.712279:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.712282:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25886720 00000020:00010000:2.0:1789760769.712287:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.712290:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.712293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.712295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.712298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.712304:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.712309:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.712321:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760769.712496:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 25882624<>25886720 00000100:00100000:3.0:1789760769.712558:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495891456:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760769.747579:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973635 gen 2 00000100:00100000:3.0:1789760769.747610:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205266a00 x1876699495891456/t4294973636(4294973636) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760785 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760769.747647:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760769.747653:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.747662:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.747683:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760769.747706:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495891456:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.747793:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6319, 6319] 00000080:00200000:2.0:1789760769.747805:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.747811:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.747822:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25886720 i_blocks 51209 00000020:00200000:2.0:1789760769.747826:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.747833:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.747869:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25886720, count: 4096 00000080:00200000:2.0:1789760769.747875:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25886720, bytes: 4096 00000080:00200000:2.0:1789760769.747882:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.747889:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.747891:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25886720, 25890816) 0 0 00000080:00200000:2.0:1789760769.747895:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.747907:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6320, 6320] 00020000:00200000:2.0:1789760769.747914:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25886720 chunk: [25886720, 25890816) 25890816, 4096 00020000:00200000:2.0:1789760769.747917:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.747932:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6320, 6320] 00000008:00010000:2.0:1789760769.747945:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.747950:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.747958:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25886720, 25890816) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.747968:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25886720=0x18b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.748012:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.748014:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25890816 00000020:00010000:2.0:1789760769.748021:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.748024:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.748027:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.748030:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.748033:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.748038:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.748043:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.748055:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760769.748147:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 25886720<>25890816 00000100:00100000:3.0:1789760769.748368:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760769.748393:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495891584:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760769.783479:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973636 gen 2 00000100:00100000:3.0:1789760769.783521:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205267b80 x1876699495891584/t4294973637(4294973637) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760785 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760769.783559:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760769.783565:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.783573:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.783597:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760769.783615:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495891584:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.784408:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6320, 6320] 00000080:00200000:2.0:1789760769.784422:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.784428:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.784439:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25890816 i_blocks 51209 00000020:00200000:2.0:1789760769.784443:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.784452:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.784479:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25890816, count: 4096 00000080:00200000:2.0:1789760769.784485:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25890816, bytes: 4096 00000080:00200000:2.0:1789760769.784494:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.784513:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.784515:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25890816, 25894912) 0 0 00000080:00200000:2.0:1789760769.784518:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.784530:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6321, 6321] 00020000:00200000:2.0:1789760769.784537:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25890816 chunk: [25890816, 25894912) 25894912, 4096 00020000:00200000:2.0:1789760769.784541:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.784557:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6321, 6321] 00000008:00010000:2.0:1789760769.784570:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.784575:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.784583:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25890816, 25894912) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.784593:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25890816=0x18b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.784640:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.784643:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25894912 00000020:00010000:2.0:1789760769.784650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.784653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.784657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.784660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.784662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.784669:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.784674:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.784689:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760769.784794:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 25890816<>25894912 00000100:00100000:2.0:1789760769.784841:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760769.784859:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495891712:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760769.823976:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973638 gen 2 00000008:00010000:2.0:1789760769.824025:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760769.824032:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760769.824040:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760769.824066:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760769.824091:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495891712:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760769.824194:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6321, 6321] 00000080:00200000:1.0:1789760769.824206:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.824216:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.824232:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25894912 i_blocks 51209 00000020:00200000:1.0:1789760769.824236:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.824244:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.824270:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25894912, count: 4096 00000080:00200000:1.0:1789760769.824277:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25894912, bytes: 4096 00000080:00200000:1.0:1789760769.824287:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.824294:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.824296:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25894912, 25899008) 0 0 00000080:00200000:1.0:1789760769.824300:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.824314:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6322, 6322] 00020000:00200000:1.0:1789760769.824320:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25894912 chunk: [25894912, 25899008) 25899008, 4096 00020000:00200000:1.0:1789760769.824325:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.824486:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6322, 6322] 00000008:00010000:1.0:1789760769.824500:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.824506:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760769.824514:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25894912, 25899008) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.824524:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25894912=0x18b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.824568:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.824571:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25899008 00000020:00010000:1.0:1789760769.824578:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.824581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.824584:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.824587:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.824589:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.824596:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.824601:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.824616:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760769.824710:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25894912<>25899008 00000100:00100000:0.0:1789760769.824821:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760769.824837:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495891840:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760769.909659:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973639 gen 2 00000008:00010000:1.0:1789760769.909697:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760769.909703:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.909712:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.909743:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760769.909767:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495891840:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.909885:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6322, 6322] 00000080:00200000:2.0:1789760769.909899:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.909906:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.909918:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25899008 i_blocks 51209 00000020:00200000:2.0:1789760769.909921:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.909929:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.909958:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25899008, count: 4096 00000080:00200000:2.0:1789760769.909965:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25899008, bytes: 4096 00000080:00200000:2.0:1789760769.909972:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.909980:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.909982:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25899008, 25903104) 0 0 00000080:00200000:2.0:1789760769.909986:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.909999:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6323, 6323] 00020000:00200000:2.0:1789760769.910007:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25899008 chunk: [25899008, 25903104) 25903104, 4096 00020000:00200000:2.0:1789760769.910011:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.910027:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6323, 6323] 00000008:00010000:2.0:1789760769.910041:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.910047:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.910056:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25899008, 25903104) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.910066:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25899008=0x18b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.910110:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.910113:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25903104 00000020:00010000:2.0:1789760769.910120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.910124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.910126:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.910131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.910133:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.910140:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.910146:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.910160:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760769.910296:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 25899008<>25903104 00000100:00100000:3.0:1789760769.910487:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760769.910502:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495891968:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760769.965783:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973640 gen 2 00000008:00010000:3.0:1789760769.965821:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760769.965828:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.965847:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.965867:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760769.965890:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495891968:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760769.965985:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6323, 6323] 00000080:00200000:2.0:1789760769.965998:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.966004:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.966015:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25903104 i_blocks 51209 00000020:00200000:2.0:1789760769.966018:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.966026:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.966055:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25903104, count: 4096 00000080:00200000:2.0:1789760769.966061:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25903104, bytes: 4096 00000080:00200000:2.0:1789760769.966068:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.966075:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.966076:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25903104, 25907200) 0 0 00000080:00200000:2.0:1789760769.966080:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.966093:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6324, 6324] 00020000:00200000:2.0:1789760769.966101:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25903104 chunk: [25903104, 25907200) 25907200, 4096 00020000:00200000:2.0:1789760769.966105:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.966121:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6324, 6324] 00000008:00010000:2.0:1789760769.966134:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.966140:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760769.966148:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25903104, 25907200) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.966159:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25903104=0x18b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.966230:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.966233:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25907200 00000020:00010000:2.0:1789760769.966239:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.966243:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.966246:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.966249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.966252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.966258:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.966263:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.966276:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760769.966367:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 25903104<>25907200 00000100:00100000:2.0:1789760769.966418:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760769.966431:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495892096:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760770.030261:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760770.030271:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.030280:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.030304:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760770.030327:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495892096:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760770.030424:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6324, 6324] 00000080:00200000:1.0:1789760770.030436:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.030441:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.030452:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25907200 i_blocks 51209 00000020:00200000:1.0:1789760770.030455:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.030463:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.030489:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25907200, count: 4096 00000080:00200000:1.0:1789760770.030504:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25907200, bytes: 4096 00000080:00200000:1.0:1789760770.030511:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.030519:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.030521:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25907200, 25911296) 0 0 00000080:00200000:1.0:1789760770.030525:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.030536:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6325, 6325] 00020000:00200000:1.0:1789760770.030542:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25907200 chunk: [25907200, 25911296) 25911296, 4096 00020000:00200000:1.0:1789760770.030546:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.030559:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6325, 6325] 00000008:00010000:1.0:1789760770.030572:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.030576:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760770.030584:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25907200, 25911296) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.030595:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25907200=0x18b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.030635:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.030637:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25911296 00000020:00010000:1.0:1789760770.030644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.030647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.030650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.030653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.030655:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.030661:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.030665:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.030678:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760770.030765:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25907200<>25911296 00000100:00100000:0.0:1789760770.031031:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495892224:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760770.108649:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973642 gen 2 00000008:00010000:1.0:1789760770.108694:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760770.108701:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.108710:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.108731:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760770.108753:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495892224:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760770.109284:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6325, 6325] 00000080:00200000:2.0:1789760770.109298:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.109304:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.109317:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25911296 i_blocks 51209 00000020:00200000:2.0:1789760770.109321:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.109330:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.109354:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25911296, count: 4096 00000080:00200000:2.0:1789760770.109361:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25911296, bytes: 4096 00000080:00200000:2.0:1789760770.109369:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.109377:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.109379:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25911296, 25915392) 0 0 00000080:00200000:2.0:1789760770.109383:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.109406:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6326, 6326] 00020000:00200000:2.0:1789760770.109414:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25911296 chunk: [25911296, 25915392) 25915392, 4096 00020000:00200000:2.0:1789760770.109417:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.109433:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6326, 6326] 00000008:00010000:2.0:1789760770.109446:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.109452:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760770.109462:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25911296, 25915392) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.109472:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25911296=0x18b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.109524:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.109527:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25915392 00000020:00010000:2.0:1789760770.109533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.109537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.109539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.109543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.109546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.109551:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.109556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.109570:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760770.109663:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 25911296<>25915392 00000100:00100000:3.0:1789760770.109808:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760770.109827:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495892352:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760770.152387:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973643 gen 2 00000008:00010000:3.0:1789760770.152531:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760770.152537:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.152546:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.152566:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760770.152591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495892352:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760770.152699:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6326, 6326] 00000080:00200000:2.0:1789760770.152712:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.152717:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.152734:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25915392 i_blocks 51209 00000020:00200000:2.0:1789760770.152738:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.152746:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.152774:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25915392, count: 4096 00000080:00200000:2.0:1789760770.152779:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25915392, bytes: 4096 00000080:00200000:2.0:1789760770.152787:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.152794:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.152797:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25915392, 25919488) 0 0 00000080:00200000:2.0:1789760770.152800:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.152813:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6327, 6327] 00020000:00200000:2.0:1789760770.152820:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25915392 chunk: [25915392, 25919488) 25919488, 4096 00020000:00200000:2.0:1789760770.152824:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.152840:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6327, 6327] 00000008:00010000:2.0:1789760770.152853:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.152859:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760770.152868:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25915392, 25919488) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.152878:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25915392=0x18b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.152923:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.152926:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25919488 00000020:00010000:2.0:1789760770.152933:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.152936:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.152939:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.152942:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.152945:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.152951:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.152956:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.152970:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760770.153062:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 25915392<>25919488 00000100:00100000:2.0:1789760770.153101:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760770.153114:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495892480:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760770.225154:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973644 gen 2 00000008:00010000:2.0:1789760770.225227:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:2.0:1789760770.225234:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.225242:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.225262:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:2.0:1789760770.225286:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495892480:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.225445:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6327, 6327] 00000080:00200000:3.0:1789760770.225459:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.225465:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.225478:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25919488 i_blocks 51209 00000020:00200000:3.0:1789760770.225481:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.225490:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.225524:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25919488, count: 4096 00000080:00200000:3.0:1789760770.225531:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25919488, bytes: 4096 00000080:00200000:3.0:1789760770.225539:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.225546:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.225549:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25919488, 25923584) 0 0 00000080:00200000:3.0:1789760770.225553:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.225566:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6328, 6328] 00020000:00200000:3.0:1789760770.225574:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25919488 chunk: [25919488, 25923584) 25923584, 4096 00020000:00200000:3.0:1789760770.225578:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.225592:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6328, 6328] 00000008:00010000:3.0:1789760770.225606:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.225612:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.225620:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25919488, 25923584) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.225630:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25919488=0x18b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.225869:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.225872:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25923584 00000020:00010000:3.0:1789760770.225879:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.225883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.225885:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.225888:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.225891:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.225897:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.225902:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.225917:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760770.226020:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 25919488<>25923584 00000100:00100000:3.0:1789760770.226059:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760770.226076:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495892608:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760770.276269:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760770.276280:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.276289:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.276311:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760770.276334:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495892608:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760770.276664:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6328, 6328] 00000080:00200000:2.0:1789760770.276681:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.276687:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.276699:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25923584 i_blocks 51209 00000020:00200000:2.0:1789760770.276703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.276711:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.276737:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25923584, count: 4096 00000080:00200000:2.0:1789760770.276743:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25923584, bytes: 4096 00000080:00200000:2.0:1789760770.276751:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.276758:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.276760:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25923584, 25927680) 0 0 00000080:00200000:2.0:1789760770.276764:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.276776:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6329, 6329] 00020000:00200000:2.0:1789760770.276784:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25923584 chunk: [25923584, 25927680) 25927680, 4096 00020000:00200000:2.0:1789760770.276788:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.276803:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6329, 6329] 00000008:00010000:2.0:1789760770.276818:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.276823:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760770.276833:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25923584, 25927680) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.276844:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25923584=0x18b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.276891:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.276894:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25927680 00000020:00010000:2.0:1789760770.276901:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.276904:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.276907:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.276910:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.276913:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.276920:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.276926:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.276941:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760770.277116:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 25923584<>25927680 00000100:00100000:2.0:1789760770.277263:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760770.277278:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495892736:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760770.348798:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973646 gen 2 00000008:00010000:2.0:1789760770.348847:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:2.0:1789760770.348854:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.348863:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.348934:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:2.0:1789760770.348959:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495892736:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.349026:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6329, 6329] 00000080:00200000:3.0:1789760770.349039:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.349044:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.349057:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25927680 i_blocks 51209 00000020:00200000:3.0:1789760770.349060:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.349068:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.349096:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25927680, count: 4096 00000080:00200000:3.0:1789760770.349102:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25927680, bytes: 4096 00000080:00200000:3.0:1789760770.349110:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.349116:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.349209:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25927680, 25931776) 0 0 00000080:00200000:3.0:1789760770.349214:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.349227:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6330, 6330] 00020000:00200000:3.0:1789760770.349233:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25927680 chunk: [25927680, 25931776) 25931776, 4096 00020000:00200000:3.0:1789760770.349238:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.349253:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6330, 6330] 00000008:00010000:3.0:1789760770.349268:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.349273:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.349283:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25927680, 25931776) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.349294:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25927680=0x18ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.349340:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.349342:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25931776 00000020:00010000:3.0:1789760770.349349:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.349352:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.349355:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.349359:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.349361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.349368:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.349373:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.349387:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760770.349488:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 25927680<>25931776 00000100:00100000:3.0:1789760770.349603:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760770.349618:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495892864:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760770.415452:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760770.415464:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.415471:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.415494:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760770.415519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495892864:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760770.415668:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6330, 6330] 00000080:00200000:1.0:1789760770.415681:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.415687:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.415698:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25931776 i_blocks 51209 00000020:00200000:1.0:1789760770.415703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.415710:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.415735:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25931776, count: 4096 00000080:00200000:1.0:1789760770.415741:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25931776, bytes: 4096 00000080:00200000:1.0:1789760770.415748:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.415755:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.415757:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25931776, 25935872) 0 0 00000080:00200000:1.0:1789760770.415761:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.415774:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6331, 6331] 00020000:00200000:1.0:1789760770.415781:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25931776 chunk: [25931776, 25935872) 25935872, 4096 00020000:00200000:1.0:1789760770.415784:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.415798:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6331, 6331] 00000008:00010000:1.0:1789760770.415812:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.415817:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760770.415826:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25931776, 25935872) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.415837:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25931776=0x18bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.415879:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.415882:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25935872 00000020:00010000:1.0:1789760770.415889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.415892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.415895:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.415898:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.415900:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.415906:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.415912:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.415925:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760770.416020:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25931776<>25935872 00000100:00100000:1.0:1789760770.416062:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760770.416075:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495892992:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760770.457293:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973648 gen 2 00000008:00010000:1.0:1789760770.457338:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760770.457344:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.457352:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.457507:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760770.457533:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495892992:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760770.458337:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6331, 6331] 00000080:00200000:0.0:1789760770.458361:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.458576:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.458596:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25935872 i_blocks 51209 00000020:00200000:0.0:1789760770.458601:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.458615:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.458651:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25935872, count: 4096 00000080:00200000:0.0:1789760770.458804:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25935872, bytes: 4096 00000080:00200000:0.0:1789760770.458817:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.458826:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.458828:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25935872, 25939968) 0 0 00000080:00200000:0.0:1789760770.458833:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.458850:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6332, 6332] 00020000:00200000:0.0:1789760770.458857:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25935872 chunk: [25935872, 25939968) 25939968, 4096 00020000:00200000:0.0:1789760770.458863:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.458885:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6332, 6332] 00000008:00010000:0.0:1789760770.458906:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.458912:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760770.458925:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25935872, 25939968) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.458941:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25935872=0x18bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.459010:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.459013:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25939968 00000020:00010000:0.0:1789760770.459035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.459039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.459042:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.459046:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.459049:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.459058:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.459064:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.459089:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760770.459258:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 25935872<>25939968 00000100:00100000:0.0:1789760770.459301:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760770.459325:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495893120:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760770.503732:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973649 gen 2 00000008:00010000:1.0:1789760770.503784:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760770.503793:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.503804:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.503827:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760770.503856:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495893120:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760770.504059:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6332, 6332] 00000080:00200000:0.0:1789760770.504077:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.504083:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.504100:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25939968 i_blocks 51209 00000020:00200000:0.0:1789760770.504104:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.504112:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.504147:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25939968, count: 4096 00000080:00200000:0.0:1789760770.504154:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25939968, bytes: 4096 00000080:00200000:0.0:1789760770.504164:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.504206:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.504209:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25939968, 25944064) 0 0 00000080:00200000:0.0:1789760770.504213:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.504230:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6333, 6333] 00020000:00200000:0.0:1789760770.504238:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25939968 chunk: [25939968, 25944064) 25944064, 4096 00020000:00200000:0.0:1789760770.504243:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.504265:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6333, 6333] 00000008:00010000:0.0:1789760770.504283:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.504288:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760770.504300:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25939968, 25944064) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.504314:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25939968=0x18bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.504382:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.504385:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25944064 00000020:00010000:0.0:1789760770.504403:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.504407:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.504410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.504414:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.504417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.504424:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.504431:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.504448:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760770.504570:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 25939968<>25944064 00000100:00100000:1.0:1789760770.504881:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760770.504903:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495893248:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760770.509466:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760770.549569:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973650 gen 2 00000008:00010000:1.0:1789760770.549621:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760770.549631:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.549643:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.549707:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760770.549744:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495893248:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760770.552547:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6333, 6333] 00000080:00200000:0.0:1789760770.552572:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.552579:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.552599:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25944064 i_blocks 51209 00000020:00200000:0.0:1789760770.552604:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.554517:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.554579:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25944064, count: 4096 00000080:00200000:0.0:1789760770.554591:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25944064, bytes: 4096 00000080:00200000:0.0:1789760770.554609:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.554627:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.554630:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25944064, 25948160) 0 0 00000080:00200000:0.0:1789760770.554637:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.554657:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6334, 6334] 00020000:00200000:0.0:1789760770.554666:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25944064 chunk: [25944064, 25948160) 25948160, 4096 00020000:00200000:0.0:1789760770.554671:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.554696:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6334, 6334] 00000008:00010000:0.0:1789760770.554715:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.554721:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760770.554734:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25944064, 25948160) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.554749:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25944064=0x18be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.558576:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.558588:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25948160 00000020:00010000:0.0:1789760770.558607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.558612:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.558618:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.558624:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.558627:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.558642:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.558654:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.558695:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760770.560274:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 25944064<>25948160 00000100:00100000:0.0:1789760770.560508:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760770.560532:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495893376:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760770.601050:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760770.601064:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.601074:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.601092:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760770.601120:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495893376:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.601216:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6334, 6334] 00000080:00200000:3.0:1789760770.601454:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.601462:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.601477:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25948160 i_blocks 51209 00000020:00200000:3.0:1789760770.601480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.601490:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.601525:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25948160, count: 4096 00000080:00200000:3.0:1789760770.601533:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25948160, bytes: 4096 00000080:00200000:3.0:1789760770.601543:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.601552:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.601555:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25948160, 25952256) 0 0 00000080:00200000:3.0:1789760770.601559:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.601577:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6335, 6335] 00020000:00200000:3.0:1789760770.601586:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25948160 chunk: [25948160, 25952256) 25952256, 4096 00020000:00200000:3.0:1789760770.601590:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.601609:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6335, 6335] 00000008:00010000:3.0:1789760770.601626:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.601633:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.601645:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25948160, 25952256) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.601660:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25948160=0x18bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.601792:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.601795:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25952256 00000020:00010000:3.0:1789760770.601805:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.601809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.601813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.601815:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.601818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.601824:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.601830:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.601845:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760770.601973:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 25948160<>25952256 00000100:00100000:2.0:1789760770.602048:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495893504:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760770.639548:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973652 gen 2 00000008:00010000:2.0:1789760770.639616:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760770.639624:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.639635:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.639659:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760770.639691:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495893504:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.639817:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6335, 6335] 00000080:00200000:3.0:1789760770.639835:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.639842:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.639858:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25952256 i_blocks 51209 00000020:00200000:3.0:1789760770.639862:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.639872:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.640028:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25952256, count: 4096 00000080:00200000:3.0:1789760770.640037:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25952256, bytes: 4096 00000080:00200000:3.0:1789760770.640047:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.640056:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.640059:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25952256, 25956352) 0 0 00000080:00200000:3.0:1789760770.640063:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.640081:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6336, 6336] 00020000:00200000:3.0:1789760770.640091:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25952256 chunk: [25952256, 25956352) 25956352, 4096 00020000:00200000:3.0:1789760770.640096:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.640115:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6336, 6336] 00000008:00010000:3.0:1789760770.640134:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.640140:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.640153:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25952256, 25956352) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.640188:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25952256=0x18c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.640242:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.640244:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25956352 00000020:00010000:3.0:1789760770.640252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.640256:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.640260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.640263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.640265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.640273:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.640280:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.640297:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760770.640491:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 25952256<>25956352 00000100:00100000:2.0:1789760770.640581:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760770.640599:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495893632:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760770.673677:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973653 gen 2 00000008:00010000:2.0:1789760770.673730:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:2.0:1789760770.673738:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.673750:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.673818:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:2.0:1789760770.673851:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495893632:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.674257:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6336, 6336] 00000080:00200000:3.0:1789760770.674275:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.674282:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.674298:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25956352 i_blocks 51209 00000020:00200000:3.0:1789760770.674302:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.674311:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.674343:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25956352, count: 4096 00000080:00200000:3.0:1789760770.674349:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25956352, bytes: 4096 00000080:00200000:3.0:1789760770.674360:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.674370:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.674441:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25956352, 25960448) 0 0 00000080:00200000:3.0:1789760770.674445:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.674463:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6337, 6337] 00020000:00200000:3.0:1789760770.674472:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25956352 chunk: [25956352, 25960448) 25960448, 4096 00020000:00200000:3.0:1789760770.674477:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.674497:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6337, 6337] 00000008:00010000:3.0:1789760770.674576:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.674583:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.674596:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25956352, 25960448) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.674611:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25956352=0x18c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.674673:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.674676:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25960448 00000020:00010000:3.0:1789760770.674685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.674689:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.674691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.674695:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.674699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.674707:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.674779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.674800:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760770.674983:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 25956352<>25960448 00000100:00100000:2.0:1789760770.675080:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495893760:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760770.708507:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:2.0:1789760770.708521:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.708532:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.708553:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:2.0:1789760770.708587:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495893760:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.708696:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6337, 6337] 00000080:00200000:3.0:1789760770.708714:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.708776:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.708793:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25960448 i_blocks 51209 00000020:00200000:3.0:1789760770.708796:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.708807:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.708841:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25960448, count: 4096 00000080:00200000:3.0:1789760770.708849:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25960448, bytes: 4096 00000080:00200000:3.0:1789760770.708857:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.708866:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.708868:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25960448, 25964544) 0 0 00000080:00200000:3.0:1789760770.708872:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.708890:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6338, 6338] 00020000:00200000:3.0:1789760770.708899:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25960448 chunk: [25960448, 25964544) 25964544, 4096 00020000:00200000:3.0:1789760770.708904:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.708924:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6338, 6338] 00000008:00010000:3.0:1789760770.708942:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.708948:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.708960:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25960448, 25964544) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.708973:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25960448=0x18c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.709038:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.709042:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25964544 00000020:00010000:3.0:1789760770.709051:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.709055:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.709058:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.709062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.709066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.709074:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.709081:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.709099:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760770.709248:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 25960448<>25964544 00000100:00100000:2.0:1789760770.709439:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760770.709456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495893888:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760770.770508:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973655 gen 2 00000008:00010000:2.0:1789760770.770577:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:2.0:1789760770.770585:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.770595:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.770618:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:2.0:1789760770.770649:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495893888:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.770860:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6338, 6338] 00000080:00200000:3.0:1789760770.770877:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.770884:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.770900:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25964544 i_blocks 51209 00000020:00200000:3.0:1789760770.770904:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.770913:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.770947:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25964544, count: 4096 00000080:00200000:3.0:1789760770.770954:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25964544, bytes: 4096 00000080:00200000:3.0:1789760770.770964:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.770974:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.770977:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25964544, 25968640) 0 0 00000080:00200000:3.0:1789760770.770981:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.770999:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6339, 6339] 00020000:00200000:3.0:1789760770.771007:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25964544 chunk: [25964544, 25968640) 25968640, 4096 00020000:00200000:3.0:1789760770.771011:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.771033:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6339, 6339] 00000008:00010000:3.0:1789760770.771050:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.771056:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.771068:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25964544, 25968640) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.771083:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25964544=0x18c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.771142:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.771144:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25968640 00000020:00010000:3.0:1789760770.771154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.771158:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.771161:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.771164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.771193:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.771202:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.771211:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.771229:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760770.771358:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 25964544<>25968640 00000100:00100000:2.0:1789760770.771690:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495894016:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760770.835233:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973656 gen 2 00000008:00010000:2.0:1789760770.835289:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:2.0:1789760770.835297:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.835307:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.835329:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:2.0:1789760770.835359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495894016:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.835559:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6339, 6339] 00000080:00200000:3.0:1789760770.835576:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.835582:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.835599:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25968640 i_blocks 51209 00000020:00200000:3.0:1789760770.835603:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.835612:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.835649:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25968640, count: 4096 00000080:00200000:3.0:1789760770.835657:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25968640, bytes: 4096 00000080:00200000:3.0:1789760770.835666:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.835675:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.835677:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25968640, 25972736) 0 0 00000080:00200000:3.0:1789760770.835681:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.835696:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6340, 6340] 00020000:00200000:3.0:1789760770.835703:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25968640 chunk: [25968640, 25972736) 25972736, 4096 00020000:00200000:3.0:1789760770.835706:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.835726:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6340, 6340] 00000008:00010000:3.0:1789760770.835744:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.835751:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.835763:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25968640, 25972736) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.835777:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25968640=0x18c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.835823:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.835825:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25972736 00000020:00010000:3.0:1789760770.835834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.835837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.835840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.835844:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.835847:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.835856:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.835990:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.836008:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760770.836137:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 25968640<>25972736 00000100:00100000:2.0:1789760770.836192:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760770.836212:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495894144:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760770.875845:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973657 gen 2 00000008:00010000:2.0:1789760770.875907:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:2.0:1789760770.875914:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760770.875925:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760770.875947:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:2.0:1789760770.875973:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495894144:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.876065:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6340, 6340] 00000080:00200000:3.0:1789760770.876082:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.876088:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.876102:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25972736 i_blocks 51209 00000020:00200000:3.0:1789760770.876105:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.876113:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.876145:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25972736, count: 4096 00000080:00200000:3.0:1789760770.876151:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25972736, bytes: 4096 00000080:00200000:3.0:1789760770.876160:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.876189:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.876191:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25972736, 25976832) 0 0 00000080:00200000:3.0:1789760770.876196:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.876211:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6341, 6341] 00020000:00200000:3.0:1789760770.876220:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25972736 chunk: [25972736, 25976832) 25976832, 4096 00020000:00200000:3.0:1789760770.876225:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.876245:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6341, 6341] 00000008:00010000:3.0:1789760770.876263:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.876270:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.876281:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25972736, 25976832) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.876295:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25972736=0x18c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.876349:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.876352:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25976832 00000020:00010000:3.0:1789760770.876362:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.876366:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.876374:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.876377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.876380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.876400:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.876407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.876424:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760770.876538:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 25972736<>25976832 00000100:00100000:3.0:1789760770.876578:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760770.876592:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495894272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760770.922148:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973658 gen 2 00000008:00010000:3.0:1789760770.922228:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:3.0:1789760770.922238:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.922249:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.922338:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:3.0:1789760770.922369:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495894272:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760770.922467:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6341, 6341] 00000080:00200000:1.0:1789760770.922485:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.922492:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.922507:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25976832 i_blocks 51209 00000020:00200000:1.0:1789760770.922511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.922520:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.922557:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25976832, count: 4096 00000080:00200000:1.0:1789760770.922564:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25976832, bytes: 4096 00000080:00200000:1.0:1789760770.922574:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.922584:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.922587:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25976832, 25980928) 0 0 00000080:00200000:1.0:1789760770.922592:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.922609:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6342, 6342] 00020000:00200000:1.0:1789760770.922618:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25976832 chunk: [25976832, 25980928) 25980928, 4096 00020000:00200000:1.0:1789760770.922624:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.922643:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6342, 6342] 00000008:00010000:1.0:1789760770.922660:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.922666:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760770.922677:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25976832, 25980928) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.922689:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25976832=0x18c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.922801:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.922805:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25980928 00000020:00010000:1.0:1789760770.922814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.922818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.922822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.922826:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.922829:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.922836:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.922843:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.922973:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760770.923095:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25976832<>25980928 00000100:00100000:0.0:1789760770.923194:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495894400:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760770.951694:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973659 gen 2 00000008:00010000:1.0:1789760770.951848:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760770.952278:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.952292:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.952310:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760770.952340:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495894400:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760770.952452:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6342, 6342] 00000080:00200000:3.0:1789760770.952469:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.952475:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.952489:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25980928 i_blocks 51209 00000020:00200000:3.0:1789760770.952493:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.952502:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.952533:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25980928, count: 4096 00000080:00200000:3.0:1789760770.952541:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25980928, bytes: 4096 00000080:00200000:3.0:1789760770.952551:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.952559:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.952562:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25980928, 25985024) 0 0 00000080:00200000:3.0:1789760770.952566:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.952581:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6343, 6343] 00020000:00200000:3.0:1789760770.952588:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25980928 chunk: [25980928, 25985024) 25985024, 4096 00020000:00200000:3.0:1789760770.952593:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.952609:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6343, 6343] 00000008:00010000:3.0:1789760770.952624:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.952629:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760770.952640:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25980928, 25985024) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.952652:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25980928=0x18c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.952693:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.952695:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25985024 00000020:00010000:3.0:1789760770.952703:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.952706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000008:00010000:1.0:1789760770.953518:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:1.0:1789760770.953663:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25980928<>25985024 00000020:00010000:3.0:1789760770.953816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.953820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.953823:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.953834:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.953842:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760770.954100:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495894528:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.003948:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973660 gen 2 00000008:00010000:1.0:1789760771.003998:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:1.0:1789760771.004007:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.004019:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.004040:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:1.0:1789760771.004068:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495894528:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760771.004212:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6343, 6343] 00000080:00200000:3.0:1789760771.004227:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.004234:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.004249:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25985024 i_blocks 51209 00000020:00200000:3.0:1789760771.004253:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.004262:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.004297:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25985024, count: 4096 00000080:00200000:3.0:1789760771.004305:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25985024, bytes: 4096 00000080:00200000:3.0:1789760771.004315:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.004324:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760771.004327:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25985024, 25989120) 0 0 00000080:00200000:3.0:1789760771.004332:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.004348:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6344, 6344] 00020000:00200000:3.0:1789760771.004357:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25985024 chunk: [25985024, 25989120) 25989120, 4096 00020000:00200000:3.0:1789760771.004361:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.004381:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6344, 6344] 00000008:00010000:3.0:1789760771.004412:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.004419:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760771.004430:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25985024, 25989120) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.004446:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25985024=0x18c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.004557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.004560:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25989120 00000020:00010000:3.0:1789760771.004569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.004572:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.004576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.004579:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.004582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.004589:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.004596:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.004612:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760771.004734:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 25985024<>25989120 00000100:00100000:2.0:1789760771.004820:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495894656:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760771.045870:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973661 gen 2 00000008:00010000:2.0:1789760771.045930:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:2.0:1789760771.045938:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760771.045950:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760771.045973:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:2.0:1789760771.045999:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495894656:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760771.046155:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6344, 6344] 00000080:00200000:1.0:1789760771.046189:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760771.046197:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760771.046215:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25989120 i_blocks 51209 00000020:00200000:1.0:1789760771.046218:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760771.046227:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760771.046275:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25989120, count: 4096 00000080:00200000:1.0:1789760771.046282:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25989120, bytes: 4096 00000080:00200000:1.0:1789760771.046293:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760771.046302:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760771.046304:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25989120, 25993216) 0 0 00000080:00200000:1.0:1789760771.046308:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760771.046323:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6345, 6345] 00020000:00200000:1.0:1789760771.046331:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25989120 chunk: [25989120, 25993216) 25993216, 4096 00020000:00200000:1.0:1789760771.046336:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760771.046357:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6345, 6345] 00000008:00010000:1.0:1789760771.046375:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760771.046380:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760771.046406:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25989120, 25993216) append 0 append lockpos 0 00000080:00200000:1.0:1789760771.046420:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25989120=0x18c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760771.046497:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760771.046501:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25993216 00000020:00010000:1.0:1789760771.046511:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760771.046514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.046517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.046521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760771.046524:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760771.046531:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760771.046538:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760771.046989:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760771.047125:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 25989120<>25993216 00000100:00100000:0.0:1789760771.047222:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495894784:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760771.107327:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760771.107345:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.107358:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.107394:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760771.107425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495894784:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760771.107515:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6345, 6345] 00000080:00200000:3.0:1789760771.107531:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.107538:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.107555:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25993216 i_blocks 51209 00000020:00200000:3.0:1789760771.107558:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.107568:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.107604:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25993216, count: 4096 00000080:00200000:3.0:1789760771.107611:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25993216, bytes: 4096 00000080:00200000:3.0:1789760771.107622:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.107631:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760771.107633:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25993216, 25997312) 0 0 00000080:00200000:3.0:1789760771.107638:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.107656:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6346, 6346] 00020000:00200000:3.0:1789760771.107664:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25993216 chunk: [25993216, 25997312) 25997312, 4096 00020000:00200000:3.0:1789760771.107669:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.107688:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6346, 6346] 00000008:00010000:3.0:1789760771.107778:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.107784:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760771.107797:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25993216, 25997312) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.107815:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25993216=0x18ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.107872:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.107875:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25997312 00000020:00010000:3.0:1789760771.107885:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.107889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.107892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.107895:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.107898:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.107906:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.107915:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.107931:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760771.108064:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 25993216<>25997312 00000100:00100000:2.0:1789760771.108659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495894912:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760771.169890:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973663 gen 2 00000008:00010000:2.0:1789760771.169955:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:2.0:1789760771.169963:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760771.169974:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760771.169999:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:2.0:1789760771.170029:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495894912:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760771.170525:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6346, 6346] 00000080:00200000:3.0:1789760771.170543:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.170550:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.170568:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25997312 i_blocks 51209 00000020:00200000:3.0:1789760771.170572:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.170582:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.170614:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25997312, count: 4096 00000080:00200000:3.0:1789760771.170622:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25997312, bytes: 4096 00000080:00200000:3.0:1789760771.170637:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.170647:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760771.170649:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25997312, 26001408) 0 0 00000080:00200000:3.0:1789760771.170654:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.170671:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6347, 6347] 00020000:00200000:3.0:1789760771.170680:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25997312 chunk: [25997312, 26001408) 26001408, 4096 00020000:00200000:3.0:1789760771.170685:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.170705:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6347, 6347] 00000008:00010000:3.0:1789760771.170725:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.170731:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760771.170743:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25997312, 26001408) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.170757:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=25997312=0x18cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.170815:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.170817:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26001408 00000020:00010000:3.0:1789760771.170825:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.170829:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.170832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.170836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.170839:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.170845:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.170852:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.170869:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760771.170992:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 25997312<>26001408 00000100:00100000:2.0:1789760771.171216:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760771.171247:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495895040:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760771.218562:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973664 gen 2 00000008:00010000:2.0:1789760771.218614:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:2.0:1789760771.218623:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760771.218635:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760771.218716:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:2.0:1789760771.218754:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495895040:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760771.218994:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6347, 6347] 00000080:00200000:3.0:1789760771.219012:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.219018:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.219162:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26001408 i_blocks 51209 00000020:00200000:2.0:1789760771.219186:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.219196:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.219232:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26001408, count: 4096 00000080:00200000:2.0:1789760771.219239:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26001408, bytes: 4096 00000080:00200000:2.0:1789760771.219250:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.219259:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.219261:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26001408, 26005504) 0 0 00000080:00200000:2.0:1789760771.219266:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.219281:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6348, 6348] 00020000:00200000:2.0:1789760771.219289:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26001408 chunk: [26001408, 26005504) 26005504, 4096 00020000:00200000:2.0:1789760771.219294:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.219312:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6348, 6348] 00000008:00010000:2.0:1789760771.219329:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.219334:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760771.219345:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26001408, 26005504) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.219358:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26001408=0x18cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.219576:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.219579:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26005504 00000020:00010000:2.0:1789760771.219588:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.219592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.219595:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.219598:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.219601:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.219608:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.219616:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.219631:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760771.220114:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26001408<>26005504 00000100:00100000:3.0:1789760771.220165:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760771.220296:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495895168:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760771.279640:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973665 gen 2 00000008:00010000:2.0:1789760771.279693:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:2.0:1789760771.279702:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760771.279713:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760771.279889:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:2.0:1789760771.279920:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495895168:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760771.280013:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6348, 6348] 00000080:00200000:3.0:1789760771.280030:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.280036:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.280052:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26005504 i_blocks 51209 00000020:00200000:3.0:1789760771.280057:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.280066:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.280130:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26005504, count: 4096 00000080:00200000:3.0:1789760771.280140:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26005504, bytes: 4096 00000080:00200000:3.0:1789760771.280151:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.280160:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760771.280163:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26005504, 26009600) 0 0 00000080:00200000:3.0:1789760771.280193:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.280212:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6349, 6349] 00020000:00200000:3.0:1789760771.280221:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26005504 chunk: [26005504, 26009600) 26009600, 4096 00020000:00200000:3.0:1789760771.280226:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.280247:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6349, 6349] 00000008:00010000:3.0:1789760771.280265:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.280271:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760771.280284:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26005504, 26009600) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.280298:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26005504=0x18cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.280631:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.280636:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26009600 00000020:00010000:3.0:1789760771.280646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.280650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.280654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.280657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.280660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.280669:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.280676:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.280697:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760771.280938:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 26005504<>26009600 00000100:00100000:2.0:1789760771.281023:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495895296:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760771.312541:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973666 gen 2 00000008:00010000:2.0:1789760771.312594:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:2.0:1789760771.312602:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760771.312613:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760771.312636:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:2.0:1789760771.312665:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495895296:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.312826:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6349, 6349] 00000080:00200000:0.0:1789760771.312843:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.312850:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.312865:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26009600 i_blocks 51209 00000020:00200000:0.0:1789760771.312869:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.312878:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.312911:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26009600, count: 4096 00000080:00200000:0.0:1789760771.312919:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26009600, bytes: 4096 00000080:00200000:0.0:1789760771.312928:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.312937:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.312939:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26009600, 26013696) 0 0 00000080:00200000:0.0:1789760771.312944:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.312960:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6350, 6350] 00020000:00200000:0.0:1789760771.312967:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26009600 chunk: [26009600, 26013696) 26013696, 4096 00020000:00200000:0.0:1789760771.312972:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.312993:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6350, 6350] 00000008:00010000:0.0:1789760771.313011:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.313017:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.313028:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26009600, 26013696) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.313044:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26009600=0x18ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.313102:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.313105:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26013696 00000020:00010000:0.0:1789760771.313115:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.313119:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.313122:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.313125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.313129:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.313135:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.313142:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.313158:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760771.313310:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 26009600<>26013696 00000100:00100000:1.0:1789760771.314047:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495895424:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760771.348257:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760771.348274:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.348285:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.349067:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760771.349159:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495895424:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.349610:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6350, 6350] 00000080:00200000:0.0:1789760771.349630:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.349637:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.349654:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26013696 i_blocks 51209 00000020:00200000:0.0:1789760771.349905:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.349919:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.350023:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26013696, count: 4096 00000080:00200000:0.0:1789760771.350033:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26013696, bytes: 4096 00000080:00200000:0.0:1789760771.350048:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.350057:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.350060:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26013696, 26017792) 0 0 00000080:00200000:0.0:1789760771.350065:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.350083:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6351, 6351] 00020000:00200000:0.0:1789760771.350091:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26013696 chunk: [26013696, 26017792) 26017792, 4096 00020000:00200000:0.0:1789760771.350096:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.350117:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6351, 6351] 00000008:00010000:0.0:1789760771.350137:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.350143:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.350155:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26013696, 26017792) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.350454:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26013696=0x18cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.351921:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.351930:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26017792 00000020:00010000:0.0:1789760771.351944:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.351949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.351953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.351956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.351959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.351978:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.351988:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.352033:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760771.352297:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 26013696<>26017792 00000100:00100000:1.0:1789760771.353429:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760771.353461:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495895552:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.399327:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973668 gen 2 00000008:00010000:1.0:1789760771.399521:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760771.399528:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.399537:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.399705:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000080:00200000:0.0:1789760771.399806:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6351, 6351] 00000080:00200000:0.0:1789760771.399821:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.399826:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.399839:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26017792 i_blocks 51209 00000020:00200000:0.0:1789760771.399843:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.399850:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.399874:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26017792, count: 4096 00000080:00200000:0.0:1789760771.399881:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26017792, bytes: 4096 00000080:00200000:0.0:1789760771.399889:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.399896:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.399898:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26017792, 26021888) 0 0 00000080:00200000:0.0:1789760771.399914:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.399927:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6352, 6352] 00020000:00200000:0.0:1789760771.399935:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26017792 chunk: [26017792, 26021888) 26021888, 4096 00020000:00200000:0.0:1789760771.399939:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.399955:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6352, 6352] 00000008:00010000:0.0:1789760771.399969:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.399974:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.399982:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26017792, 26021888) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.399993:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26017792=0x18d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.400034:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.400038:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26021888 00000020:00010000:0.0:1789760771.400044:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.400048:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.400050:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.400052:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.400055:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.400060:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.400065:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760771.400118:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495895552:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760771.400152:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:1.0:1789760771.400276:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26017792<>26021888 00000100:00100000:1.0:1789760771.400321:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495895680:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.439640:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973669 gen 2 00000008:00010000:1.0:1789760771.439751:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760771.439759:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.439767:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.439787:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760771.439811:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495895680:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.440424:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6352, 6352] 00000080:00200000:0.0:1789760771.440508:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.440514:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.440527:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26021888 i_blocks 51209 00000020:00200000:0.0:1789760771.440531:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.440540:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.440567:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26021888, count: 4096 00000080:00200000:0.0:1789760771.440574:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26021888, bytes: 4096 00000080:00200000:0.0:1789760771.440582:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.440590:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.440592:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26021888, 26025984) 0 0 00000080:00200000:0.0:1789760771.440596:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.440608:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6353, 6353] 00020000:00200000:0.0:1789760771.440615:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26021888 chunk: [26021888, 26025984) 26025984, 4096 00020000:00200000:0.0:1789760771.440619:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.440635:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6353, 6353] 00000008:00010000:0.0:1789760771.440649:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.440655:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.440665:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26021888, 26025984) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.440676:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26021888=0x18d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.441458:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.441463:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26025984 00000020:00010000:0.0:1789760771.441473:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.441476:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.441479:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.441483:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.441486:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.441494:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.441500:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.441519:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760771.441622:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 26021888<>26025984 00000100:00100000:1.0:1789760771.442442:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495895808:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.487707:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973670 gen 2 00000008:00010000:1.0:1789760771.487745:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760771.487752:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.487761:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.487782:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760771.487807:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495895808:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.487966:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6353, 6353] 00000080:00200000:0.0:1789760771.487978:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.487983:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.487996:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26025984 i_blocks 51209 00000020:00200000:0.0:1789760771.487999:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.488008:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.488034:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26025984, count: 4096 00000080:00200000:0.0:1789760771.488040:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26025984, bytes: 4096 00000080:00200000:0.0:1789760771.488103:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.488112:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.488114:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26025984, 26030080) 0 0 00000080:00200000:0.0:1789760771.488118:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.488131:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6354, 6354] 00020000:00200000:0.0:1789760771.488138:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26025984 chunk: [26025984, 26030080) 26030080, 4096 00020000:00200000:0.0:1789760771.488142:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.488157:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6354, 6354] 00000008:00010000:0.0:1789760771.488201:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.488207:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.488216:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26025984, 26030080) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.488227:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26025984=0x18d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.488279:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.488282:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26030080 00000020:00010000:0.0:1789760771.488290:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.488293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.488295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.488298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.488300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.488306:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.488312:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.488326:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760771.488558:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 26025984<>26030080 00000100:00100000:1.0:1789760771.488667:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760771.488683:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495895936:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.524162:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973671 gen 2 00000008:00010000:1.0:1789760771.524226:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760771.524233:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.524242:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.524300:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760771.524325:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495895936:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.524468:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6354, 6354] 00000080:00200000:0.0:1789760771.524481:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.524487:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.524499:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26030080 i_blocks 51209 00000020:00200000:0.0:1789760771.524502:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.524511:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.524537:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26030080, count: 4096 00000080:00200000:0.0:1789760771.524543:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26030080, bytes: 4096 00000080:00200000:0.0:1789760771.524552:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.524559:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.524561:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26030080, 26034176) 0 0 00000080:00200000:0.0:1789760771.524565:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.524578:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6355, 6355] 00020000:00200000:0.0:1789760771.524586:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26030080 chunk: [26030080, 26034176) 26034176, 4096 00020000:00200000:0.0:1789760771.524589:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.524605:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6355, 6355] 00000008:00010000:0.0:1789760771.524620:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.524625:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.524634:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26030080, 26034176) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.524645:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26030080=0x18d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.524690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.524693:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26034176 00000020:00010000:0.0:1789760771.524700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.524704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.524706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.524709:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.524712:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.524719:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.524724:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.524739:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760771.524972:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 26030080<>26034176 00000100:00100000:1.0:1789760771.525036:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495896064:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760771.532752:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760771.560223:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973672 gen 2 00000008:00010000:1.0:1789760771.560257:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760771.560263:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.560271:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.560287:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760771.560311:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495896064:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.560408:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6355, 6355] 00000080:00200000:0.0:1789760771.560420:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.560425:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.560436:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26034176 i_blocks 51209 00000020:00200000:0.0:1789760771.560440:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.560446:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.560469:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26034176, count: 4096 00000080:00200000:0.0:1789760771.560474:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26034176, bytes: 4096 00000080:00200000:0.0:1789760771.560481:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.560488:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.560490:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26034176, 26038272) 0 0 00000080:00200000:0.0:1789760771.560494:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.560505:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6356, 6356] 00020000:00200000:0.0:1789760771.560512:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26034176 chunk: [26034176, 26038272) 26038272, 4096 00020000:00200000:0.0:1789760771.560516:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.560530:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6356, 6356] 00000008:00010000:0.0:1789760771.560542:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.560547:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.560556:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26034176, 26038272) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.560568:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26034176=0x18d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.560614:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.560617:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26038272 00000020:00010000:0.0:1789760771.560623:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.560626:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.560629:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.560632:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.560635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.560640:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.560644:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.560656:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760771.560814:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 26034176<>26038272 00000100:00100000:1.0:1789760771.560982:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896192:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.594222:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973673 gen 2 00000008:00010000:1.0:1789760771.594259:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760771.594266:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.594274:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.594290:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760771.594313:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896192:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.594467:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6356, 6356] 00000080:00200000:0.0:1789760771.594479:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.594485:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.594497:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26038272 i_blocks 51209 00000020:00200000:0.0:1789760771.594500:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.594508:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.594533:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26038272, count: 4096 00000080:00200000:0.0:1789760771.594539:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26038272, bytes: 4096 00000080:00200000:0.0:1789760771.594547:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.594554:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.594556:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26038272, 26042368) 0 0 00000080:00200000:0.0:1789760771.594560:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.594573:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6357, 6357] 00020000:00200000:0.0:1789760771.594580:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26038272 chunk: [26038272, 26042368) 26042368, 4096 00020000:00200000:0.0:1789760771.594584:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.594598:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6357, 6357] 00000008:00010000:0.0:1789760771.594613:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.594619:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.594627:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26038272, 26042368) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.594637:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26038272=0x18d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.594678:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.594681:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26042368 00000020:00010000:0.0:1789760771.594687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.594691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.594693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.594696:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.594699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.594705:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.594778:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.594794:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760771.594895:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 26038272<>26042368 00000100:00100000:1.0:1789760771.595040:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760771.595054:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896320:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.650331:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973674 gen 2 00000008:00010000:1.0:1789760771.650380:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760771.650395:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.650405:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.650422:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760771.650446:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896320:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.650842:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6357, 6357] 00000080:00200000:0.0:1789760771.650854:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.650860:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.650872:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26042368 i_blocks 51209 00000020:00200000:0.0:1789760771.650875:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.650883:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.650908:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26042368, count: 4096 00000080:00200000:0.0:1789760771.650915:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26042368, bytes: 4096 00000080:00200000:0.0:1789760771.650923:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.650930:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.650933:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26042368, 26046464) 0 0 00000080:00200000:0.0:1789760771.650937:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.650949:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6358, 6358] 00020000:00200000:0.0:1789760771.650956:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26042368 chunk: [26042368, 26046464) 26046464, 4096 00020000:00200000:0.0:1789760771.650961:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.650976:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6358, 6358] 00000008:00010000:0.0:1789760771.650990:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.650995:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.651005:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26042368, 26046464) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.651017:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26042368=0x18d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.651064:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.651068:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26046464 00000020:00010000:0.0:1789760771.651074:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.651077:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.651080:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.651084:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.651086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.651092:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.651097:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.651111:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760771.651233:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 26042368<>26046464 00000100:00100000:1.0:1789760771.651488:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896448:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760771.681594:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760771.681605:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.681614:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.681633:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760771.681660:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896448:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.681979:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6358, 6358] 00000080:00200000:0.0:1789760771.681992:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.681998:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.682010:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26046464 i_blocks 51209 00000020:00200000:0.0:1789760771.682013:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.682020:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.682046:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26046464, count: 4096 00000080:00200000:0.0:1789760771.682052:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26046464, bytes: 4096 00000080:00200000:0.0:1789760771.682060:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.682067:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.682069:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26046464, 26050560) 0 0 00000080:00200000:0.0:1789760771.682073:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.682087:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6359, 6359] 00020000:00200000:0.0:1789760771.682095:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26046464 chunk: [26046464, 26050560) 26050560, 4096 00020000:00200000:0.0:1789760771.682099:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.682115:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6359, 6359] 00000008:00010000:0.0:1789760771.682129:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.682134:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.682143:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26046464, 26050560) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.682160:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26046464=0x18d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.682232:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.682235:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26050560 00000020:00010000:0.0:1789760771.682243:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.682246:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.682249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.682252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.682255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.682261:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.682266:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.682281:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760771.682572:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 26046464<>26050560 00000100:00100000:1.0:1789760771.682638:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495896576:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.762362:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973675 gen 2 00000100:00100000:1.0:1789760771.763506:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211834a80 x1876699495896576/t4294973676(4294973676) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760787 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760771.763558:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760771.763564:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.763572:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.763594:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760771.763611:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495896576:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.763828:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6359, 6359] 00000080:00200000:0.0:1789760771.763840:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.763845:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.763856:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26050560 i_blocks 51209 00000020:00200000:0.0:1789760771.763859:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.763867:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.763891:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26050560, count: 4096 00000080:00200000:0.0:1789760771.764528:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26050560, bytes: 4096 00000080:00200000:0.0:1789760771.764541:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.764548:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.764550:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26050560, 26054656) 0 0 00000080:00200000:0.0:1789760771.764554:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.764567:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6360, 6360] 00020000:00200000:0.0:1789760771.764574:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26050560 chunk: [26050560, 26054656) 26054656, 4096 00020000:00200000:0.0:1789760771.764578:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.764595:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6360, 6360] 00000008:00010000:0.0:1789760771.764610:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.764617:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.764626:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26050560, 26054656) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.764635:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26050560=0x18d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.764683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.764686:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26054656 00000020:00010000:0.0:1789760771.764692:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.764696:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.764699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.764702:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.764704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.764709:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.764714:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.764730:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760771.766025:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 26050560<>26054656 00000100:00100000:1.0:1789760771.766198:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896704:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.814976:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973677 gen 2 00000008:00010000:1.0:1789760771.815026:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760771.815033:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.815040:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.815059:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760771.815083:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896704:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.815197:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6360, 6360] 00000080:00200000:0.0:1789760771.815212:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.815218:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.815230:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26054656 i_blocks 51209 00000020:00200000:0.0:1789760771.815234:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.815242:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.815267:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26054656, count: 4096 00000080:00200000:0.0:1789760771.816448:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26054656, bytes: 4096 00000080:00200000:0.0:1789760771.816467:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.816476:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.816478:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26054656, 26058752) 0 0 00000080:00200000:0.0:1789760771.816483:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.816499:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6361, 6361] 00020000:00200000:0.0:1789760771.816507:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26054656 chunk: [26054656, 26058752) 26058752, 4096 00020000:00200000:0.0:1789760771.816512:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.816531:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6361, 6361] 00000008:00010000:0.0:1789760771.816546:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.816552:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.816563:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26054656, 26058752) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.816575:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26054656=0x18d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.816631:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.816633:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26058752 00000020:00010000:0.0:1789760771.816641:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.816644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.816647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.816650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.816653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.816660:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.816666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.816689:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760771.817480:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 26054656<>26058752 00000100:00100000:1.0:1789760771.817881:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495896832:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.864367:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973678 gen 2 00000008:00010000:1.0:1789760771.864413:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760771.864420:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.864428:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.864490:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760771.864513:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495896832:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.865397:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6361, 6361] 00000080:00200000:0.0:1789760771.865412:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.865418:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.865431:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26058752 i_blocks 51209 00000020:00200000:0.0:1789760771.865435:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.865445:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.865470:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26058752, count: 4096 00000080:00200000:0.0:1789760771.865476:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26058752, bytes: 4096 00000080:00200000:0.0:1789760771.865487:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.865496:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.865498:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26058752, 26062848) 0 0 00000080:00200000:0.0:1789760771.865502:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.865514:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6362, 6362] 00020000:00200000:0.0:1789760771.865522:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26058752 chunk: [26058752, 26062848) 26062848, 4096 00020000:00200000:0.0:1789760771.865526:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.865541:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6362, 6362] 00000008:00010000:0.0:1789760771.865555:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.865561:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.865570:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26058752, 26062848) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.865580:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26058752=0x18da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.865631:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.865634:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26062848 00000020:00010000:0.0:1789760771.865641:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.865644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.865646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.865649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.865652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.865659:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.865665:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.865684:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760771.865851:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 26058752<>26062848 00000100:00100000:1.0:1789760771.865977:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896960:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.906783:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973679 gen 2 00000008:00010000:1.0:1789760771.906821:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760771.906828:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.906837:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.906855:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760771.906879:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495896960:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.906999:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6362, 6362] 00000080:00200000:0.0:1789760771.907021:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.907026:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.907044:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26062848 i_blocks 51209 00000020:00200000:0.0:1789760771.907047:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.907055:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.907081:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26062848, count: 4096 00000080:00200000:0.0:1789760771.907086:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26062848, bytes: 4096 00000080:00200000:0.0:1789760771.907095:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.907101:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.907103:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26062848, 26066944) 0 0 00000080:00200000:0.0:1789760771.907218:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.907238:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6363, 6363] 00020000:00200000:0.0:1789760771.907245:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26062848 chunk: [26062848, 26066944) 26066944, 4096 00020000:00200000:0.0:1789760771.907249:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.907273:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6363, 6363] 00000008:00010000:0.0:1789760771.907293:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.907299:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.907307:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26062848, 26066944) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.907326:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26062848=0x18db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.907371:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.907374:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26066944 00000020:00010000:0.0:1789760771.907394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.907398:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.907401:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.907404:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.907406:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.907412:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.907418:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.907432:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760771.907550:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26062848<>26066944 00000100:00100000:1.0:1789760771.907625:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897088:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.947567:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973680 gen 2 00000008:00010000:1.0:1789760771.947615:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760771.947623:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.947634:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.947653:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760771.947680:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897088:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.947937:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6363, 6363] 00000080:00200000:0.0:1789760771.947954:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.947961:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.947979:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26066944 i_blocks 51209 00000020:00200000:0.0:1789760771.947983:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.947993:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.948026:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26066944, count: 4096 00000080:00200000:0.0:1789760771.948033:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26066944, bytes: 4096 00000080:00200000:0.0:1789760771.948043:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.948052:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.948055:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26066944, 26071040) 0 0 00000080:00200000:0.0:1789760771.948069:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.948088:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6364, 6364] 00020000:00200000:0.0:1789760771.948096:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26066944 chunk: [26066944, 26071040) 26071040, 4096 00020000:00200000:0.0:1789760771.948101:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.948128:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6364, 6364] 00000008:00010000:0.0:1789760771.948145:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.948150:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.948162:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26066944, 26071040) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.948201:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26066944=0x18dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.948256:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.948259:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26071040 00000020:00010000:0.0:1789760771.948267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.948270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.948272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.948276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.948278:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.948285:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.948291:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.948308:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760771.948991:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 26066944<>26071040 00000100:00100000:1.0:1789760771.949148:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495897216:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760771.980557:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973681 gen 2 00000008:00010000:1.0:1789760771.980618:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760771.980627:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.980639:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.980653:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760771.980687:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495897216:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760771.984647:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6364, 6364] 00000080:00200000:0.0:1789760771.984678:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.984686:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.984714:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26071040 i_blocks 51209 00000020:00200000:0.0:1789760771.984720:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.984736:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.984771:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26071040, count: 4096 00000080:00200000:0.0:1789760771.984780:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26071040, bytes: 4096 00000080:00200000:0.0:1789760771.984796:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.984807:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.984809:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26071040, 26075136) 0 0 00000080:00200000:0.0:1789760771.984814:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.984831:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6365, 6365] 00020000:00200000:0.0:1789760771.984840:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26071040 chunk: [26071040, 26075136) 26075136, 4096 00020000:00200000:0.0:1789760771.984845:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.984871:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6365, 6365] 00000008:00010000:0.0:1789760771.984892:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.984897:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760771.984909:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26071040, 26075136) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.984926:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26071040=0x18dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.984999:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.985002:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26075136 00000020:00010000:0.0:1789760771.985011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.985014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.985018:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.985022:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.985025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.985033:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.985040:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.985063:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760771.985256:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 26071040<>26075136 00000100:00100000:1.0:1789760771.985962:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760771.985992:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495897344:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760772.017358:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973682 gen 2 00000008:00010000:1.0:1789760772.017424:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760772.017433:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.017444:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.017508:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760772.017536:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495897344:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760772.018085:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6365, 6365] 00000080:00200000:0.0:1789760772.018105:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.018111:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.018128:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26075136 i_blocks 51209 00000020:00200000:0.0:1789760772.018131:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.018142:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.018200:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26075136, count: 4096 00000080:00200000:0.0:1789760772.018208:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26075136, bytes: 4096 00000080:00200000:0.0:1789760772.018219:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.018227:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.018229:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26075136, 26079232) 0 0 00000080:00200000:0.0:1789760772.018234:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.018249:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6366, 6366] 00020000:00200000:0.0:1789760772.018258:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26075136 chunk: [26075136, 26079232) 26079232, 4096 00020000:00200000:0.0:1789760772.018263:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.018285:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6366, 6366] 00000008:00010000:0.0:1789760772.018303:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.018308:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760772.018319:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26075136, 26079232) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.018336:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26075136=0x18de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.018523:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.018526:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26079232 00000020:00010000:0.0:1789760772.018535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.018539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.018542:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.018546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.018549:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.018557:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.018564:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.018582:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760772.018771:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 26075136<>26079232 00000100:00100000:1.0:1789760772.018839:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760772.018861:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897472:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760772.060943:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973683 gen 2 00000008:00010000:1.0:1789760772.061000:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760772.061011:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.061022:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.061053:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760772.061086:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897472:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760772.061152:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6366, 6366] 00000080:00200000:0.0:1789760772.061204:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.061211:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.061274:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26079232 i_blocks 51209 00000020:00200000:0.0:1789760772.061279:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.061288:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.061322:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26079232, count: 4096 00000080:00200000:0.0:1789760772.061329:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26079232, bytes: 4096 00000080:00200000:0.0:1789760772.061341:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.061349:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.061351:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26079232, 26083328) 0 0 00000080:00200000:0.0:1789760772.061356:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.061373:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6367, 6367] 00020000:00200000:0.0:1789760772.061390:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26079232 chunk: [26079232, 26083328) 26083328, 4096 00020000:00200000:0.0:1789760772.061395:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.061416:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6367, 6367] 00000008:00010000:0.0:1789760772.061436:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.061442:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760772.061454:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26079232, 26083328) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.061468:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26079232=0x18df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.061518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.061521:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26083328 00000020:00010000:0.0:1789760772.061530:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.061533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.061536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.061540:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.061644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.061654:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.061661:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.061674:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760772.061929:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 26079232<>26083328 00000100:00100000:1.0:1789760772.062303:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897600:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760772.106636:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973684 gen 2 00000008:00010000:1.0:1789760772.106686:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760772.106695:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.106771:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.106800:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760772.106837:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897600:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760772.106963:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6367, 6367] 00000080:00200000:2.0:1789760772.106980:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.106987:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.107003:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26083328 i_blocks 51209 00000020:00200000:2.0:1789760772.107007:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.107017:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.107052:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26083328, count: 4096 00000080:00200000:2.0:1789760772.107059:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26083328, bytes: 4096 00000080:00200000:2.0:1789760772.107070:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.107079:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.107081:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26083328, 26087424) 0 0 00000080:00200000:2.0:1789760772.107086:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.107101:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6368, 6368] 00020000:00200000:2.0:1789760772.107110:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26083328 chunk: [26083328, 26087424) 26087424, 4096 00020000:00200000:2.0:1789760772.107115:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.107135:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6368, 6368] 00000008:00010000:2.0:1789760772.107153:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.107159:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760772.107223:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26083328, 26087424) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.107239:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26083328=0x18e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.107294:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.107297:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26087424 00000020:00010000:2.0:1789760772.107307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.107310:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.107314:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.107317:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.107320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.107328:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.107335:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.107347:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760772.107495:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26083328<>26087424 00000100:00100000:2.0:1789760772.107542:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760772.107561:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495897728:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.141968:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973685 gen 2 00000008:00010000:2.0:1789760772.142028:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:2.0:1789760772.142037:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.142048:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.142075:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:2.0:1789760772.142108:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495897728:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760772.142130:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6368, 6368] 00000080:00200000:1.0:1789760772.142148:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.142154:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.142193:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26087424 i_blocks 51209 00000020:00200000:1.0:1789760772.142198:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.142209:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.142244:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26087424, count: 4096 00000080:00200000:1.0:1789760772.142252:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26087424, bytes: 4096 00000080:00200000:1.0:1789760772.142263:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.142272:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.142274:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26087424, 26091520) 0 0 00000080:00200000:1.0:1789760772.142278:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.142293:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6369, 6369] 00020000:00200000:1.0:1789760772.142301:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26087424 chunk: [26087424, 26091520) 26091520, 4096 00020000:00200000:1.0:1789760772.142306:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.142326:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6369, 6369] 00000008:00010000:1.0:1789760772.142343:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.142348:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760772.142360:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26087424, 26091520) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.142374:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26087424=0x18e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.142442:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.142444:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26091520 00000020:00010000:1.0:1789760772.142463:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.142467:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.142471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.142474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.142477:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.142484:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.142490:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.142506:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760772.142640:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26087424<>26091520 00000100:00100000:1.0:1789760772.142687:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760772.142703:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897856:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760772.171412:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973686 gen 2 00000008:00010000:1.0:1789760772.171464:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760772.171473:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.171484:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.171506:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760772.171534:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897856:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760772.171691:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6369, 6369] 00000080:00200000:0.0:1789760772.171708:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.171715:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.171730:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26091520 i_blocks 51209 00000020:00200000:0.0:1789760772.171734:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.171744:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.171779:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26091520, count: 4096 00000080:00200000:0.0:1789760772.171787:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26091520, bytes: 4096 00000080:00200000:0.0:1789760772.171798:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.171807:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.171810:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26091520, 26095616) 0 0 00000080:00200000:0.0:1789760772.171814:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.171832:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6370, 6370] 00020000:00200000:0.0:1789760772.171841:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26091520 chunk: [26091520, 26095616) 26095616, 4096 00020000:00200000:0.0:1789760772.171846:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.171868:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6370, 6370] 00000008:00010000:0.0:1789760772.171886:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.171893:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760772.171904:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26091520, 26095616) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.171919:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26091520=0x18e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.171980:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.171983:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26095616 00000020:00010000:0.0:1789760772.171992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.171995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.171998:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.172002:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.172005:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.172012:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.172019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.172035:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760772.172153:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 26091520<>26095616 00000100:00100000:1.0:1789760772.172235:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897984:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760772.218508:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760772.218526:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.218538:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.218603:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760772.218635:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495897984:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760772.219844:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6370, 6370] 00000080:00200000:0.0:1789760772.219940:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.219947:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.219965:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26095616 i_blocks 51209 00000020:00200000:0.0:1789760772.219971:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.219983:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.220018:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26095616, count: 4096 00000080:00200000:0.0:1789760772.220027:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26095616, bytes: 4096 00000080:00200000:0.0:1789760772.220041:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.220051:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.220053:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26095616, 26099712) 0 0 00000080:00200000:0.0:1789760772.220058:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.220077:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6371, 6371] 00020000:00200000:0.0:1789760772.220085:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26095616 chunk: [26095616, 26099712) 26099712, 4096 00020000:00200000:0.0:1789760772.220090:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.220110:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6371, 6371] 00000008:00010000:0.0:1789760772.220125:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.220130:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760772.220140:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26095616, 26099712) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.220153:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26095616=0x18e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.220247:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.220251:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26099712 00000020:00010000:0.0:1789760772.220259:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.220263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.220266:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.220269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.220272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.220280:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.220287:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.220311:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760772.220520:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 26095616<>26099712 00000100:00100000:1.0:1789760772.220619:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495898112:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760772.264988:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973688 gen 2 00000008:00010000:1.0:1789760772.265055:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760772.265063:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.265073:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.265152:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760772.265232:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495898112:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760772.265580:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6371, 6371] 00000080:00200000:0.0:1789760772.265598:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.265604:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.265620:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26099712 i_blocks 51209 00000020:00200000:0.0:1789760772.265623:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.265634:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.265665:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26099712, count: 4096 00000080:00200000:0.0:1789760772.265672:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26099712, bytes: 4096 00000080:00200000:0.0:1789760772.265685:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.265695:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.265697:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26099712, 26103808) 0 0 00000080:00200000:0.0:1789760772.265833:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.265849:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6372, 6372] 00020000:00200000:0.0:1789760772.265857:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26099712 chunk: [26099712, 26103808) 26103808, 4096 00020000:00200000:0.0:1789760772.265862:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.265883:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6372, 6372] 00000008:00010000:0.0:1789760772.265900:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.265906:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760772.265917:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26099712, 26103808) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.265931:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26099712=0x18e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.265993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.265996:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26103808 00000020:00010000:0.0:1789760772.266004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.266008:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.266011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.266014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.266017:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.266024:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.266030:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.266045:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760772.266163:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 26099712<>26103808 00000100:00100000:1.0:1789760772.266298:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495898240:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760772.310942:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973689 gen 2 00000008:00010000:1.0:1789760772.310993:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760772.311002:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.311014:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.311037:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760772.311066:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495898240:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.311407:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6372, 6372] 00000080:00200000:3.0:1789760772.311425:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.311432:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.311450:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26103808 i_blocks 51209 00000020:00200000:3.0:1789760772.311456:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.311466:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.311502:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26103808, count: 4096 00000080:00200000:3.0:1789760772.311510:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26103808, bytes: 4096 00000080:00200000:3.0:1789760772.311521:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.311530:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.311532:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26103808, 26107904) 0 0 00000080:00200000:3.0:1789760772.311536:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.311555:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6373, 6373] 00020000:00200000:3.0:1789760772.311564:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26103808 chunk: [26103808, 26107904) 26107904, 4096 00020000:00200000:3.0:1789760772.311569:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.311589:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6373, 6373] 00000008:00010000:3.0:1789760772.311607:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.311613:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.311625:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26103808, 26107904) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.311640:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26103808=0x18e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.311699:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.311703:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26107904 00000020:00010000:3.0:1789760772.311712:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.311716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.311719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.311723:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.311726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.311734:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.311741:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.311759:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760772.311899:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 26103808<>26107904 00000100:00100000:3.0:1789760772.311950:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.311969:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495898368:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760772.346496:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973690 gen 2 00000008:00010000:3.0:1789760772.346547:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760772.346555:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.346566:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.346597:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760772.346628:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495898368:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760772.346796:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6373, 6373] 00000080:00200000:2.0:1789760772.346814:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.346820:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.346837:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26107904 i_blocks 51209 00000020:00200000:2.0:1789760772.346840:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.346849:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.346884:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26107904, count: 4096 00000080:00200000:2.0:1789760772.346891:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26107904, bytes: 4096 00000080:00200000:2.0:1789760772.346901:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.346910:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.346913:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26107904, 26112000) 0 0 00000080:00200000:2.0:1789760772.346918:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.346935:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6374, 6374] 00020000:00200000:2.0:1789760772.346944:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26107904 chunk: [26107904, 26112000) 26112000, 4096 00020000:00200000:2.0:1789760772.346950:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.346969:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6374, 6374] 00000008:00010000:2.0:1789760772.346988:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.346994:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760772.347006:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26107904, 26112000) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.347020:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26107904=0x18e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.347081:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.347083:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26112000 00000020:00010000:2.0:1789760772.347093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.347096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.347100:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.347103:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.347106:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.347114:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.347121:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.347137:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760772.347304:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26107904<>26112000 00000100:00100000:2.0:1789760772.347342:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760772.347374:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495898496:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.387001:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973691 gen 2 00000008:00010000:2.0:1789760772.387057:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:2.0:1789760772.387067:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.387080:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.387105:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:2.0:1789760772.387137:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495898496:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.387277:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6374, 6374] 00000080:00200000:3.0:1789760772.387294:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.387300:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.387317:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26112000 i_blocks 51209 00000020:00200000:3.0:1789760772.387320:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.387331:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.387390:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26112000, count: 4096 00000080:00200000:3.0:1789760772.387398:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26112000, bytes: 4096 00000080:00200000:3.0:1789760772.387410:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.387419:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.387421:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26112000, 26116096) 0 0 00000080:00200000:3.0:1789760772.387425:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.387441:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6375, 6375] 00020000:00200000:3.0:1789760772.387450:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26112000 chunk: [26112000, 26116096) 26116096, 4096 00020000:00200000:3.0:1789760772.387454:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.387473:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6375, 6375] 00000008:00010000:3.0:1789760772.387491:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.387498:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.387510:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26112000, 26116096) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.387524:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26112000=0x18e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.387586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.387589:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26116096 00000020:00010000:3.0:1789760772.387599:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.387603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.387606:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.387610:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.387613:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.387621:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.387628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.387644:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760772.387905:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 26112000<>26116096 00000100:00100000:3.0:1789760772.387949:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.387967:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495898624:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760772.443540:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760772.443556:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.443567:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.443593:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760772.443623:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495898624:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760772.443636:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6375, 6375] 00000080:00200000:2.0:1789760772.443656:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.443662:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.443677:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26116096 i_blocks 51209 00000020:00200000:2.0:1789760772.443683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.443692:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.443734:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26116096, count: 4096 00000080:00200000:2.0:1789760772.443740:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26116096, bytes: 4096 00000080:00200000:2.0:1789760772.443748:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.443756:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.443759:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26116096, 26120192) 0 0 00000080:00200000:2.0:1789760772.443763:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.443782:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6376, 6376] 00020000:00200000:2.0:1789760772.443791:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26116096 chunk: [26116096, 26120192) 26120192, 4096 00020000:00200000:2.0:1789760772.443795:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.443815:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6376, 6376] 00000008:00010000:2.0:1789760772.443833:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.443838:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760772.443849:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26116096, 26120192) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.443863:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26116096=0x18e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.443918:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.443920:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26120192 00000020:00010000:2.0:1789760772.443928:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.443931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.443934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.443938:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.443940:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.443948:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.443956:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.443973:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760772.444096:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26116096<>26120192 00000100:00100000:2.0:1789760772.444139:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760772.444155:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495898752:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.484745:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973693 gen 2 00000008:00010000:2.0:1789760772.484811:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:2.0:1789760772.484819:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.484835:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.484856:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:2.0:1789760772.484885:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495898752:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.484961:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6376, 6376] 00000080:00200000:3.0:1789760772.484978:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.484985:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.485001:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26120192 i_blocks 51209 00000020:00200000:3.0:1789760772.485004:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.485014:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.485048:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26120192, count: 4096 00000080:00200000:3.0:1789760772.485056:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26120192, bytes: 4096 00000080:00200000:3.0:1789760772.485066:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.485076:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.485078:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26120192, 26124288) 0 0 00000080:00200000:3.0:1789760772.485083:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.485100:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6377, 6377] 00020000:00200000:3.0:1789760772.485108:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26120192 chunk: [26120192, 26124288) 26124288, 4096 00020000:00200000:3.0:1789760772.485112:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.485131:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6377, 6377] 00000008:00010000:3.0:1789760772.485147:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.485153:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.485192:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26120192, 26124288) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.485208:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26120192=0x18e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.485268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.485271:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26124288 00000020:00010000:3.0:1789760772.485280:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.485284:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.485288:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.485294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.485297:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.485304:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.485312:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.485329:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760772.485471:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 26120192<>26124288 00000100:00100000:3.0:1789760772.485513:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.485528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495898880:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760772.519011:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973694 gen 2 00000008:00010000:3.0:1789760772.519069:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:3.0:1789760772.519078:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.519089:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.519193:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:3.0:1789760772.519230:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495898880:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760772.519330:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6377, 6377] 00000080:00200000:2.0:1789760772.519350:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.519428:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.519444:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26124288 i_blocks 51209 00000020:00200000:2.0:1789760772.519448:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.519458:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.519492:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26124288, count: 4096 00000080:00200000:2.0:1789760772.519500:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26124288, bytes: 4096 00000080:00200000:2.0:1789760772.519512:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.519521:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.519523:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26124288, 26128384) 0 0 00000080:00200000:2.0:1789760772.519528:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.519604:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6378, 6378] 00020000:00200000:2.0:1789760772.519613:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26124288 chunk: [26124288, 26128384) 26128384, 4096 00020000:00200000:2.0:1789760772.519618:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.519641:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6378, 6378] 00000008:00010000:2.0:1789760772.519661:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.519668:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760772.519681:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26124288, 26128384) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.519755:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26124288=0x18ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.519817:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.519820:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26128384 00000020:00010000:2.0:1789760772.519831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.519834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.519838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.519842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.519845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.519989:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.519998:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.520017:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088139c0 00000008:00100000:2.0:1789760772.520148:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26124288<>26128384 00000100:00100000:3.0:1789760772.520224:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495899008:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760772.555450:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760772.572429:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973695 gen 2 00000008:00010000:3.0:1789760772.572489:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088139c0 00000020:00200000:3.0:1789760772.572497:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.572508:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.572534:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088139c0 00000100:00100000:3.0:1789760772.572564:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495899008:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760772.572680:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6378, 6378] 00000080:00200000:1.0:1789760772.572697:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.572710:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.572726:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26128384 i_blocks 51209 00000020:00200000:1.0:1789760772.572729:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.572738:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.572772:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26128384, count: 4096 00000080:00200000:1.0:1789760772.572779:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26128384, bytes: 4096 00000080:00200000:1.0:1789760772.572788:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.572796:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.572798:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26128384, 26132480) 0 0 00000080:00200000:1.0:1789760772.572803:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.572818:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6379, 6379] 00020000:00200000:1.0:1789760772.572826:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26128384 chunk: [26128384, 26132480) 26132480, 4096 00020000:00200000:1.0:1789760772.572831:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.572851:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6379, 6379] 00000008:00010000:1.0:1789760772.572868:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.572873:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760772.572884:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26128384, 26132480) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.572897:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26128384=0x18eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.572952:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.572954:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26132480 00000020:00010000:1.0:1789760772.572962:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.572966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.572969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.572973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.572975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.572982:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.572988:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.573003:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760772.573131:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26128384<>26132480 00000100:00100000:1.0:1789760772.573215:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760772.573232:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495899136:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760772.627558:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973696 gen 2 00000008:00010000:1.0:1789760772.627608:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760772.627616:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.627627:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.627656:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760772.627688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495899136:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760772.627795:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6379, 6379] 00000080:00200000:2.0:1789760772.627814:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.627821:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.627837:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26132480 i_blocks 51209 00000020:00200000:2.0:1789760772.627841:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.627851:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.629036:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26132480, count: 4096 00000080:00200000:2.0:1789760772.629049:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26132480, bytes: 4096 00000080:00200000:2.0:1789760772.629068:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.629077:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.629079:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26132480, 26136576) 0 0 00000080:00200000:2.0:1789760772.629085:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.629103:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6380, 6380] 00020000:00200000:2.0:1789760772.629112:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26132480 chunk: [26132480, 26136576) 26136576, 4096 00020000:00200000:2.0:1789760772.629117:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.629139:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6380, 6380] 00000008:00010000:2.0:1789760772.629158:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.629163:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760772.629192:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26132480, 26136576) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.629221:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26132480=0x18ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.629286:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.629289:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26136576 00000020:00010000:2.0:1789760772.629298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.629301:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.629304:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.629309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.629311:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.629320:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.629327:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.629346:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760772.630157:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26132480<>26136576 00000100:00100000:2.0:1789760772.630360:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495899264:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760772.661280:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760772.661297:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.661309:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.661418:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760772.661457:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495899264:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.662218:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6380, 6380] 00000080:00200000:3.0:1789760772.662237:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.662243:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.662261:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26136576 i_blocks 51209 00000020:00200000:3.0:1789760772.662265:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.662278:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.662315:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26136576, count: 4096 00000080:00200000:3.0:1789760772.662323:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26136576, bytes: 4096 00000080:00200000:3.0:1789760772.662337:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.662346:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.662348:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26136576, 26140672) 0 0 00000080:00200000:3.0:1789760772.662353:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.663822:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6381, 6381] 00020000:00200000:3.0:1789760772.663842:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26136576 chunk: [26136576, 26140672) 26140672, 4096 00020000:00200000:3.0:1789760772.663847:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.663885:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6381, 6381] 00000008:00010000:3.0:1789760772.663908:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.663915:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.663927:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26136576, 26140672) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.663943:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26136576=0x18ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.664014:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.664017:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26140672 00000020:00010000:3.0:1789760772.664025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.664028:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.664032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.664035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.664038:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.664048:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.664056:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.664085:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760772.667295:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 26136576<>26140672 00000100:00100000:2.0:1789760772.667408:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760772.667623:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495899392:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.703359:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973698 gen 2 00000008:00010000:2.0:1789760772.703745:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760772.703754:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.703765:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.703886:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760772.703919:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495899392:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760772.704002:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6381, 6381] 00000080:00200000:1.0:1789760772.704028:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.704035:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.704085:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26140672 i_blocks 51209 00000020:00200000:1.0:1789760772.704089:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.704099:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.704211:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26140672, count: 4096 00000080:00200000:1.0:1789760772.704219:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26140672, bytes: 4096 00000080:00200000:1.0:1789760772.704229:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.704237:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.704239:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26140672, 26144768) 0 0 00000080:00200000:1.0:1789760772.704243:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.704258:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6382, 6382] 00020000:00200000:1.0:1789760772.704265:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26140672 chunk: [26140672, 26144768) 26144768, 4096 00020000:00200000:1.0:1789760772.704269:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.704286:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6382, 6382] 00000008:00010000:1.0:1789760772.704301:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.704305:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760772.704314:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26140672, 26144768) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.704325:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26140672=0x18ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.704369:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.704371:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26144768 00000020:00010000:1.0:1789760772.704391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.704395:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.704399:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.704403:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.704406:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.704476:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.704484:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.704500:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760772.704623:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26140672<>26144768 00000100:00100000:0.0:1789760772.704740:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760772.704760:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495899520:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760772.751154:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973699 gen 2 00000008:00010000:1.0:1789760772.751242:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760772.751251:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.751261:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.751341:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760772.751561:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495899520:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760772.751649:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6382, 6382] 00000080:00200000:2.0:1789760772.751666:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.751673:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.751691:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26144768 i_blocks 51209 00000020:00200000:2.0:1789760772.751755:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.751767:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.751802:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26144768, count: 4096 00000080:00200000:2.0:1789760772.751809:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26144768, bytes: 4096 00000080:00200000:2.0:1789760772.751821:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.751831:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.751833:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26144768, 26148864) 0 0 00000080:00200000:2.0:1789760772.751838:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.751910:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6383, 6383] 00020000:00200000:2.0:1789760772.751919:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26144768 chunk: [26144768, 26148864) 26148864, 4096 00020000:00200000:2.0:1789760772.751924:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.751945:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6383, 6383] 00000008:00010000:2.0:1789760772.751963:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.751968:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760772.751980:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26144768, 26148864) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.751995:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26144768=0x18ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.752054:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.752057:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26148864 00000020:00010000:2.0:1789760772.752065:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.752069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.752072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.752076:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.752079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.752087:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.752094:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.752112:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760772.753240:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26144768<>26148864 00000100:00100000:2.0:1789760772.753310:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495899648:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.791745:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973700 gen 2 00000008:00010000:2.0:1789760772.791796:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760772.791805:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.791814:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.791835:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760772.791861:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495899648:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.791953:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6383, 6383] 00000080:00200000:3.0:1789760772.791970:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.791976:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.791992:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26148864 i_blocks 51209 00000020:00200000:3.0:1789760772.791995:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.792005:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.792049:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26148864, count: 4096 00000080:00200000:3.0:1789760772.792057:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26148864, bytes: 4096 00000080:00200000:3.0:1789760772.792067:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.792076:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.792078:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26148864, 26152960) 0 0 00000080:00200000:3.0:1789760772.792083:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.792107:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6384, 6384] 00020000:00200000:3.0:1789760772.792115:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26148864 chunk: [26148864, 26152960) 26152960, 4096 00020000:00200000:3.0:1789760772.792120:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.792138:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6384, 6384] 00000008:00010000:3.0:1789760772.792154:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.792159:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.792196:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26148864, 26152960) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.792211:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26148864=0x18f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.792264:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.792267:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26152960 00000020:00010000:3.0:1789760772.792275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.792278:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.792281:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.792285:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.792287:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.792294:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.792300:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.792317:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760772.792448:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 26148864<>26152960 00000100:00100000:2.0:1789760772.792615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495899776:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760772.830469:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:2.0:1789760772.830485:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.830496:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.830517:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:2.0:1789760772.830547:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495899776:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.830925:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6384, 6384] 00000080:00200000:3.0:1789760772.830942:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.830948:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.830969:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26152960 i_blocks 51209 00000020:00200000:3.0:1789760772.830972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.830981:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.831015:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26152960, count: 4096 00000080:00200000:3.0:1789760772.831022:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26152960, bytes: 4096 00000080:00200000:2.0:1789760772.831420:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.831492:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.831494:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26152960, 26157056) 0 0 00000080:00200000:2.0:1789760772.831500:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.831518:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6385, 6385] 00020000:00200000:2.0:1789760772.831527:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26152960 chunk: [26152960, 26157056) 26157056, 4096 00020000:00200000:2.0:1789760772.831532:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.831557:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6385, 6385] 00000008:00010000:2.0:1789760772.831578:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.831584:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760772.831595:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26152960, 26157056) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.831610:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26152960=0x18f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.831668:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.831670:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26157056 00000020:00010000:2.0:1789760772.831679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.831682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.831684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.831688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.831690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.831772:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.831780:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.831830:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760772.833823:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26152960<>26157056 00000100:00100000:2.0:1789760772.834042:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495899904:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.901304:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973701 gen 2 00000100:00100000:2.0:1789760772.901346:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264a80 x1876699495899904/t4294973702(4294973702) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760788 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760772.901412:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760772.901420:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.901430:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.901451:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760772.901476:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495899904:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.901578:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6385, 6385] 00000080:00200000:3.0:1789760772.901594:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.901600:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.901626:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26157056 i_blocks 51209 00000020:00200000:3.0:1789760772.901631:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.901640:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.901821:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26157056, count: 4096 00000080:00200000:3.0:1789760772.901830:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26157056, bytes: 4096 00000080:00200000:3.0:1789760772.901841:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.901859:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.901861:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26157056, 26161152) 0 0 00000080:00200000:3.0:1789760772.901865:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.901883:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6386, 6386] 00020000:00200000:3.0:1789760772.901893:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26157056 chunk: [26157056, 26161152) 26161152, 4096 00020000:00200000:3.0:1789760772.901897:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.901918:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6386, 6386] 00000008:00010000:3.0:1789760772.901937:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.901943:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.901953:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26157056, 26161152) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.901967:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26157056=0x18f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.902028:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.902032:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26161152 00000020:00010000:3.0:1789760772.902040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.902043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.902046:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.902049:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.902052:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.902061:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.902068:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.902085:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760772.902363:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 26157056<>26161152 00000100:00100000:2.0:1789760772.902775:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900032:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.933683:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973703 gen 2 00000008:00010000:2.0:1789760772.933750:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760772.933757:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.933766:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.933785:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760772.933811:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900032:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.933944:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6386, 6386] 00000080:00200000:3.0:1789760772.933960:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.933966:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.933980:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26161152 i_blocks 51209 00000020:00200000:3.0:1789760772.933984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.933992:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.934023:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26161152, count: 4096 00000080:00200000:3.0:1789760772.934030:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26161152, bytes: 4096 00000080:00200000:3.0:1789760772.934040:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.934048:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.934050:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26161152, 26165248) 0 0 00000080:00200000:3.0:1789760772.934054:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.934072:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6387, 6387] 00020000:00200000:3.0:1789760772.934080:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26161152 chunk: [26161152, 26165248) 26165248, 4096 00020000:00200000:3.0:1789760772.934084:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.934104:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6387, 6387] 00000008:00010000:3.0:1789760772.934121:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.934127:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.934139:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26161152, 26165248) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.934153:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26161152=0x18f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.934243:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.934247:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26165248 00000020:00010000:3.0:1789760772.934255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.934258:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.934262:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.934265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.934269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.934276:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.934284:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.934301:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760772.934436:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 26161152<>26165248 00000100:00100000:2.0:1789760772.934680:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760772.934704:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900160:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.961013:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973704 gen 2 00000008:00010000:2.0:1789760772.961058:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:2.0:1789760772.961065:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.961074:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.961100:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000080:00200000:3.0:1789760772.961120:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6387, 6387] 00000080:00200000:3.0:1789760772.961138:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.961145:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.961161:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26165248 i_blocks 51209 00000020:00200000:3.0:1789760772.961165:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000100:00100000:2.0:1789760772.961262:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900160:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.961262:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.961298:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26165248, count: 4096 00000080:00200000:3.0:1789760772.961305:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26165248, bytes: 4096 00000080:00200000:3.0:1789760772.961314:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.961323:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.961325:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26165248, 26169344) 0 0 00000080:00200000:3.0:1789760772.961330:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.961349:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6388, 6388] 00020000:00200000:3.0:1789760772.961357:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26165248 chunk: [26165248, 26169344) 26169344, 4096 00020000:00200000:3.0:1789760772.961362:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.961393:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6388, 6388] 00000008:00010000:3.0:1789760772.961411:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.961416:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.961428:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26165248, 26169344) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.961444:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26165248=0x18f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.961500:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.961504:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26169344 00000020:00010000:3.0:1789760772.961513:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.961517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.961520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.961523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.961526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.961534:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.961541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.961556:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760772.961681:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 26165248<>26169344 00000100:00100000:2.0:1789760772.961780:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900288:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760772.984413:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973705 gen 2 00000008:00010000:2.0:1789760772.984464:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760772.984473:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760772.984495:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760772.984515:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000080:00200000:3.0:1789760772.984541:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6388, 6388] 00000100:00100000:2.0:1789760772.984552:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900288:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760772.984561:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.984569:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.984584:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26169344 i_blocks 51209 00000020:00200000:3.0:1789760772.984588:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.984598:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.984631:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26169344, count: 4096 00000080:00200000:3.0:1789760772.984637:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26169344, bytes: 4096 00000080:00200000:3.0:1789760772.984646:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.984654:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.984657:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26169344, 26173440) 0 0 00000080:00200000:3.0:1789760772.984661:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.984677:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6389, 6389] 00020000:00200000:3.0:1789760772.984686:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26169344 chunk: [26169344, 26173440) 26173440, 4096 00020000:00200000:3.0:1789760772.984691:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.984724:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6389, 6389] 00000008:00010000:3.0:1789760772.984742:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.984748:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760772.984759:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26169344, 26173440) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.984773:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26169344=0x18f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.984831:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.984834:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26173440 00000020:00010000:3.0:1789760772.984855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.984859:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.984862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.984865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.984868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.984876:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.984882:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.984899:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760772.985026:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 26169344<>26173440 00000100:00100000:2.0:1789760772.985074:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495900416:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760773.018893:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760773.018910:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.018922:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.019898:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760773.019937:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495900416:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.020587:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6389, 6389] 00000080:00200000:3.0:1789760773.020607:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.020613:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.020631:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26173440 i_blocks 51209 00000020:00200000:3.0:1789760773.020637:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.020648:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.020684:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26173440, count: 4096 00000080:00200000:3.0:1789760773.020691:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26173440, bytes: 4096 00000080:00200000:3.0:1789760773.020705:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.020713:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.020716:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26173440, 26177536) 0 0 00000080:00200000:3.0:1789760773.020720:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.020738:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6390, 6390] 00020000:00200000:3.0:1789760773.020747:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26173440 chunk: [26173440, 26177536) 26177536, 4096 00020000:00200000:3.0:1789760773.020751:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.020772:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6390, 6390] 00000008:00010000:3.0:1789760773.020791:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.020796:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.020807:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26173440, 26177536) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.020825:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26173440=0x18f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.021527:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.021533:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26177536 00000020:00010000:3.0:1789760773.021544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.021549:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.021552:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.021556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.021559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.021570:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.021579:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.021609:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760773.021754:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 26173440<>26177536 00000100:00100000:2.0:1789760773.021854:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900544:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.065024:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973707 gen 2 00000008:00010000:2.0:1789760773.065099:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760773.065108:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.065118:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.065140:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760773.065208:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900544:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.065680:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6390, 6390] 00000080:00200000:3.0:1789760773.065700:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.065706:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.065722:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26177536 i_blocks 51209 00000020:00200000:3.0:1789760773.065726:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.065736:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.065772:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26177536, count: 4096 00000080:00200000:3.0:1789760773.065779:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26177536, bytes: 4096 00000080:00200000:3.0:1789760773.065792:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.065800:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.065803:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26177536, 26181632) 0 0 00000080:00200000:3.0:1789760773.065807:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.065824:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6391, 6391] 00020000:00200000:3.0:1789760773.065833:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26177536 chunk: [26177536, 26181632) 26181632, 4096 00020000:00200000:3.0:1789760773.065838:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.065860:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6391, 6391] 00000008:00010000:3.0:1789760773.065880:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.065886:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.065897:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26177536, 26181632) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.065911:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26177536=0x18f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.065966:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.065970:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26181632 00000020:00010000:3.0:1789760773.065979:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.065983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.065986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.065989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.065993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.066001:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.066007:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.066024:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760773.067671:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 26177536<>26181632 00000100:00100000:2.0:1789760773.069302:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760773.069338:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900672:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.106696:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973708 gen 2 00000008:00010000:2.0:1789760773.106747:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760773.106756:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.106766:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.107363:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760773.107408:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900672:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760773.107537:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6391, 6391] 00000080:00200000:1.0:1789760773.107553:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760773.107559:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760773.107581:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26181632 i_blocks 51209 00000020:00200000:1.0:1789760773.107586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760773.107596:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760773.107842:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26181632, count: 4096 00000080:00200000:1.0:1789760773.107850:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26181632, bytes: 4096 00000080:00200000:1.0:1789760773.107862:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760773.107871:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760773.107874:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26181632, 26185728) 0 0 00000080:00200000:1.0:1789760773.107878:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760773.107899:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6392, 6392] 00020000:00200000:1.0:1789760773.107907:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26181632 chunk: [26181632, 26185728) 26185728, 4096 00020000:00200000:1.0:1789760773.107912:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760773.107932:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6392, 6392] 00000008:00010000:1.0:1789760773.107951:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760773.107969:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760773.107980:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26181632, 26185728) append 0 append lockpos 0 00000080:00200000:1.0:1789760773.107994:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26181632=0x18f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760773.108051:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760773.108054:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26185728 00000020:00010000:1.0:1789760773.108062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760773.108066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760773.108069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760773.108072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760773.108075:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760773.108082:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760773.108089:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760773.108106:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760773.108262:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26181632<>26185728 00000100:00100000:0.0:1789760773.108423:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495900800:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760773.152367:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973709 gen 2 00000008:00010000:1.0:1789760773.152429:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760773.152437:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.152448:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.152470:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760773.152502:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495900800:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.152606:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6392, 6392] 00000080:00200000:3.0:1789760773.152622:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.152628:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.152644:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26185728 i_blocks 51209 00000020:00200000:3.0:1789760773.152648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.152656:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.152688:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26185728, count: 4096 00000080:00200000:3.0:1789760773.152694:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26185728, bytes: 4096 00000080:00200000:3.0:1789760773.152753:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.152761:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.152762:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26185728, 26189824) 0 0 00000080:00200000:3.0:1789760773.152766:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.152780:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6393, 6393] 00020000:00200000:3.0:1789760773.152787:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26185728 chunk: [26185728, 26189824) 26189824, 4096 00020000:00200000:3.0:1789760773.152791:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.152808:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6393, 6393] 00000008:00010000:3.0:1789760773.152822:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.152826:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.152836:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26185728, 26189824) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.152849:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26185728=0x18f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.152906:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.152910:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26189824 00000020:00010000:3.0:1789760773.152917:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.152921:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.152924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.152928:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.152931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.152938:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.152944:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.152960:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760773.153080:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 26185728<>26189824 00000100:00100000:2.0:1789760773.153149:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900928:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.200091:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973710 gen 2 00000008:00010000:2.0:1789760773.200142:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760773.200151:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.200162:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.200222:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760773.200253:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495900928:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.200354:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6393, 6393] 00000080:00200000:3.0:1789760773.200379:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.200387:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.200403:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26189824 i_blocks 51209 00000020:00200000:3.0:1789760773.200407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.200416:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.200448:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26189824, count: 4096 00000080:00200000:3.0:1789760773.200455:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26189824, bytes: 4096 00000080:00200000:3.0:1789760773.200465:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.200474:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.200477:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26189824, 26193920) 0 0 00000080:00200000:3.0:1789760773.200482:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.200499:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6394, 6394] 00020000:00200000:3.0:1789760773.200508:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26189824 chunk: [26189824, 26193920) 26193920, 4096 00020000:00200000:3.0:1789760773.200513:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.200534:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6394, 6394] 00000008:00010000:3.0:1789760773.200552:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.200557:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.200569:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26189824, 26193920) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.200583:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26189824=0x18fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.200637:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.200641:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26193920 00000020:00010000:3.0:1789760773.200648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.200652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.200656:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.200660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.200663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.200671:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.200677:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.200828:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760773.201006:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 26189824<>26193920 00000100:00100000:2.0:1789760773.201078:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495901056:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.234494:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973711 gen 2 00000008:00010000:2.0:1789760773.234543:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760773.234553:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.234564:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.234586:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760773.234617:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495901056:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.234639:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6394, 6394] 00000080:00200000:3.0:1789760773.234659:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.234666:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.234683:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26193920 i_blocks 51209 00000020:00200000:3.0:1789760773.234687:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.234708:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.234758:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26193920, count: 4096 00000080:00200000:3.0:1789760773.234766:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26193920, bytes: 4096 00000080:00200000:3.0:1789760773.234775:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.234784:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.234786:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26193920, 26198016) 0 0 00000080:00200000:3.0:1789760773.234790:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.234808:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6395, 6395] 00020000:00200000:3.0:1789760773.234816:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26193920 chunk: [26193920, 26198016) 26198016, 4096 00020000:00200000:3.0:1789760773.234822:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.234841:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6395, 6395] 00000008:00010000:3.0:1789760773.234859:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.234865:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.234876:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26193920, 26198016) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.234890:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26193920=0x18fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.234946:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.234950:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26198016 00000020:00010000:3.0:1789760773.235518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.235522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.235525:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.235529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.235532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.235540:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.235548:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.235565:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760773.235692:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 26193920<>26198016 00000100:00100000:2.0:1789760773.235936:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495901184:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.280736:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973712 gen 2 00000008:00010000:2.0:1789760773.280791:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760773.280799:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.280811:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.280841:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:2.0:1789760773.280884:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495901184:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.281513:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6395, 6395] 00000080:00200000:3.0:1789760773.281532:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.281539:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.281556:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26198016 i_blocks 51209 00000020:00200000:3.0:1789760773.281560:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.281569:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.281602:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26198016, count: 4096 00000080:00200000:3.0:1789760773.281609:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26198016, bytes: 4096 00000080:00200000:3.0:1789760773.281624:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.281633:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.281635:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26198016, 26202112) 0 0 00000080:00200000:3.0:1789760773.281640:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.281656:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6396, 6396] 00020000:00200000:3.0:1789760773.281664:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26198016 chunk: [26198016, 26202112) 26202112, 4096 00020000:00200000:3.0:1789760773.281669:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.281689:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6396, 6396] 00000008:00010000:3.0:1789760773.281707:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.281713:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.281725:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26198016, 26202112) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.281740:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26198016=0x18fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.281799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.281803:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26202112 00000020:00010000:3.0:1789760773.281812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.281816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.281819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.281823:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.281825:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.281833:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.281840:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.281859:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760773.281984:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 26198016<>26202112 00000100:00100000:2.0:1789760773.282590:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495901312:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.320579:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973713 gen 2 00000008:00010000:2.0:1789760773.320628:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760773.320637:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.320648:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.320672:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760773.320701:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495901312:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.321041:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6396, 6396] 00000080:00200000:3.0:1789760773.321068:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.321074:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.321089:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26202112 i_blocks 51209 00000020:00200000:3.0:1789760773.321093:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.321103:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.321134:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26202112, count: 4096 00000080:00200000:3.0:1789760773.321142:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26202112, bytes: 4096 00000080:00200000:3.0:1789760773.321153:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.321162:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.321165:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26202112, 26206208) 0 0 00000080:00200000:3.0:1789760773.321191:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.321209:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6397, 6397] 00020000:00200000:3.0:1789760773.321217:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26202112 chunk: [26202112, 26206208) 26206208, 4096 00020000:00200000:3.0:1789760773.321222:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.321241:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6397, 6397] 00000008:00010000:3.0:1789760773.321258:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.321264:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.321275:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26202112, 26206208) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.321289:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26202112=0x18fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.321345:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.321501:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:2.0:1789760773.321617:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26202112<>26206208 00000100:00100000:2.0:1789760773.321660:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495901440:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.322220:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26206208 00000020:00010000:3.0:1789760773.322235:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.322239:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.322243:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.322247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.322249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.322262:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.322270:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.358067:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:2.0:1789760773.358082:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.358094:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.358116:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:2.0:1789760773.358144:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495901440:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.358256:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6397, 6397] 00000080:00200000:3.0:1789760773.358272:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.358278:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.358293:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26206208 i_blocks 51209 00000020:00200000:3.0:1789760773.358297:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.358305:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.358337:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26206208, count: 4096 00000080:00200000:3.0:1789760773.358344:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26206208, bytes: 4096 00000080:00200000:3.0:1789760773.358419:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.358429:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.358431:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26206208, 26210304) 0 0 00000080:00200000:3.0:1789760773.358436:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.358451:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6398, 6398] 00020000:00200000:3.0:1789760773.358458:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26206208 chunk: [26206208, 26210304) 26210304, 4096 00020000:00200000:3.0:1789760773.358463:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.358484:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6398, 6398] 00000008:00010000:3.0:1789760773.358501:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.358507:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.358518:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26206208, 26210304) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.358531:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26206208=0x18fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.358586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.358589:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26210304 00000020:00010000:3.0:1789760773.358597:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.358601:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.358604:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.358607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.358610:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.358617:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.358624:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.358639:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760773.358827:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 26206208<>26210304 00000100:00100000:2.0:1789760773.358980:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760773.358997:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495901568:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.392912:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973715 gen 2 00000008:00010000:2.0:1789760773.392973:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:2.0:1789760773.392981:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.392991:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.393014:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:2.0:1789760773.393041:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495901568:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.393210:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6398, 6398] 00000080:00200000:3.0:1789760773.393227:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.393234:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.393250:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26210304 i_blocks 51209 00000020:00200000:3.0:1789760773.393254:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.393264:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.393298:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26210304, count: 4096 00000080:00200000:3.0:1789760773.393306:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26210304, bytes: 4096 00000080:00200000:3.0:1789760773.393316:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.393326:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.393329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26210304, 26214400) 0 0 00000080:00200000:3.0:1789760773.393333:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.393361:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6399, 6399] 00020000:00200000:3.0:1789760773.393377:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26210304 chunk: [26210304, 26214400) 26214400, 4096 00020000:00200000:3.0:1789760773.393382:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.393402:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6399, 6399] 00000008:00010000:3.0:1789760773.393420:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.393426:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760773.393438:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26210304, 26214400) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.393453:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26210304=0x18ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.393511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.393514:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26214400 00000020:00010000:3.0:1789760773.393523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.393527:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.393531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.393535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.393538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.393545:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.393552:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.393569:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760773.393691:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 26210304<>26214400 00000100:00100000:2.0:1789760773.393861:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760773.393880:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495901696:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.421911:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973716 gen 2 00000008:00010000:2.0:1789760773.421961:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760773.421969:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.421980:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.422034:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760773.422062:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495901696:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760773.422612:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6399, 6399] 00000080:00200000:3.0:1789760773.422629:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.422634:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.422776:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26214400 i_blocks 51209 00000020:00200000:2.0:1789760773.422781:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.422791:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.422821:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26214400, count: 4096 00000080:00200000:2.0:1789760773.422827:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26214400, bytes: 4096 00000080:00200000:2.0:1789760773.422838:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.422846:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.422896:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26214400, 26218496) 0 0 00000080:00200000:2.0:1789760773.422901:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.422918:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6400, 6400] 00020000:00200000:2.0:1789760773.422926:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26214400 chunk: [26214400, 26218496) 26218496, 4096 00020000:00200000:2.0:1789760773.422930:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.422950:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6400, 6400] 00000008:00010000:2.0:1789760773.422968:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.422973:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760773.422984:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26214400, 26218496) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.422997:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26214400=0x1900000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.423049:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.423051:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26218496 00000020:00010000:2.0:1789760773.423060:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.423063:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.423066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.423069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.423072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.423080:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.423086:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.423133:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760773.423267:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26214400<>26218496 00000100:00100000:3.0:1789760773.423329:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495901824:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760773.457250:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973717 gen 2 00000008:00010000:2.0:1789760773.457301:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760773.457309:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.457320:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.457553:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760773.457583:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495901824:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760773.459818:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6400, 6400] 00000080:00200000:1.0:1789760773.459844:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760773.460151:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760773.460187:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26218496 i_blocks 51209 00000020:00200000:1.0:1789760773.460192:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760773.460208:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760773.460244:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26218496, count: 4096 00000080:00200000:1.0:1789760773.460253:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26218496, bytes: 4096 00000080:00200000:1.0:1789760773.460267:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760773.460276:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760773.460279:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26218496, 26222592) 0 0 00000080:00200000:1.0:1789760773.460283:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760773.460301:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6401, 6401] 00020000:00200000:1.0:1789760773.460309:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26218496 chunk: [26218496, 26222592) 26222592, 4096 00020000:00200000:1.0:1789760773.460314:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760773.460336:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6401, 6401] 00000008:00010000:1.0:1789760773.460437:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760773.460444:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760773.460457:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26218496, 26222592) append 0 append lockpos 0 00000080:00200000:1.0:1789760773.460473:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26218496=0x1901000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760773.460543:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760773.460546:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26222592 00000020:00010000:1.0:1789760773.460555:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760773.460559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760773.460562:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760773.460565:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760773.460567:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760773.460576:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760773.460583:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760773.460614:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760773.462147:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26218496<>26222592 00000100:00100000:0.0:1789760773.462303:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495901952:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760773.501804:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760773.501820:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.501831:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.501852:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760773.501883:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495901952:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760773.502026:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6401, 6401] 00000080:00200000:0.0:1789760773.502043:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.502049:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.502065:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26222592 i_blocks 53257 00000020:00200000:0.0:1789760773.502068:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.502078:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.502114:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26222592, count: 4096 00000080:00200000:0.0:1789760773.502122:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26222592, bytes: 4096 00000080:00200000:0.0:1789760773.502135:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.502145:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.502147:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26222592, 26226688) 0 0 00000080:00200000:0.0:1789760773.502152:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.502209:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6402, 6402] 00020000:00200000:0.0:1789760773.502218:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26222592 chunk: [26222592, 26226688) 26226688, 4096 00020000:00200000:0.0:1789760773.502223:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.502244:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6402, 6402] 00000008:00010000:0.0:1789760773.502265:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.502271:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760773.502282:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26222592, 26226688) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.502296:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26222592=0x1902000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.502354:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.502357:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26226688 00000020:00010000:0.0:1789760773.502365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.502379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.502383:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.502386:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.502389:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.502397:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.502404:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.502420:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760773.502656:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 26222592<>26226688 00000100:00100000:1.0:1789760773.502829:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495902080:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760773.541084:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973719 gen 2 00000008:00010000:1.0:1789760773.541145:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760773.541153:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.541163:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.541211:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000080:00200000:0.0:1789760773.541259:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6402, 6402] 00000080:00200000:0.0:1789760773.541278:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.541284:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:1.0:1789760773.541293:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495902080:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760773.541300:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26226688 i_blocks 53257 00000020:00200000:0.0:1789760773.541304:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.541313:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.541348:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26226688, count: 4096 00000080:00200000:0.0:1789760773.541355:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26226688, bytes: 4096 00000080:00200000:0.0:1789760773.541364:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.541384:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.541386:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26226688, 26230784) 0 0 00000080:00200000:0.0:1789760773.541390:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.541409:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6403, 6403] 00020000:00200000:0.0:1789760773.541416:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26226688 chunk: [26226688, 26230784) 26230784, 4096 00020000:00200000:0.0:1789760773.541421:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.541445:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6403, 6403] 00000008:00010000:0.0:1789760773.541465:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.541470:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760773.541481:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26226688, 26230784) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.541496:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26226688=0x1903000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.541552:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.541555:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26230784 00000020:00010000:0.0:1789760773.541563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.541567:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.541569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.541572:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.541575:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.541583:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.541590:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.541605:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760773.541796:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 26226688<>26230784 00000100:00100000:1.0:1789760773.541869:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760773.541885:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495902208:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760773.573482:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973720 gen 2 00000008:00010000:1.0:1789760773.573540:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760773.573549:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.573561:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.573617:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760773.573648:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495902208:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760773.574061:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6403, 6403] 00000080:00200000:0.0:1789760773.574079:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.574086:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.574103:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26230784 i_blocks 53257 00000020:00200000:0.0:1789760773.574108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.574118:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.574151:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26230784, count: 4096 00000080:00200000:0.0:1789760773.574158:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26230784, bytes: 4096 00000080:00200000:0.0:1789760773.574206:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.574216:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.574218:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26230784, 26234880) 0 0 00000080:00200000:0.0:1789760773.574222:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.574241:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6404, 6404] 00020000:00200000:0.0:1789760773.574303:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26230784 chunk: [26230784, 26234880) 26234880, 4096 00020000:00200000:0.0:1789760773.574308:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.574329:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6404, 6404] 00000008:00010000:0.0:1789760773.574347:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.574354:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760773.574423:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26230784, 26234880) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.574439:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26230784=0x1904000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.574556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.574559:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26234880 00000020:00010000:0.0:1789760773.574568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.574571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.574575:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.574579:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.574582:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.574590:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.574597:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.574615:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760773.574793:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 26230784<>26234880 00000100:00100000:1.0:1789760773.574864:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495902336:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760773.579983:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760773.614789:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973721 gen 2 00000008:00010000:1.0:1789760773.614838:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760773.614846:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.614858:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.614877:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760773.614907:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495902336:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760773.615009:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6404, 6404] 00000080:00200000:0.0:1789760773.615026:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.615033:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.615049:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26234880 i_blocks 53257 00000020:00200000:0.0:1789760773.615053:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.615063:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.615099:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26234880, count: 4096 00000080:00200000:0.0:1789760773.615107:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26234880, bytes: 4096 00000080:00200000:0.0:1789760773.615116:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.615126:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.615128:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26234880, 26238976) 0 0 00000080:00200000:0.0:1789760773.615135:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.615151:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6405, 6405] 00020000:00200000:0.0:1789760773.615160:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26234880 chunk: [26234880, 26238976) 26238976, 4096 00020000:00200000:0.0:1789760773.615202:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.615222:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6405, 6405] 00000008:00010000:0.0:1789760773.615240:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.615246:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760773.615257:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26234880, 26238976) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.615272:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26234880=0x1905000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.615329:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.615332:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26238976 00000020:00010000:0.0:1789760773.615341:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.615345:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.615348:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.615351:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.615354:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.615361:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.615378:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.615394:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760773.615690:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 26234880<>26238976 00000100:00100000:1.0:1789760773.615831:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760773.615849:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495902464:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760773.645936:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973722 gen 2 00000008:00010000:1.0:1789760773.645985:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760773.645993:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.646003:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.646014:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760773.646041:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495902464:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760773.646696:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6405, 6405] 00000080:00200000:0.0:1789760773.646716:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.646723:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.646741:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26238976 i_blocks 53257 00000020:00200000:0.0:1789760773.646745:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.646754:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.646986:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26238976, count: 4096 00000080:00200000:0.0:1789760773.646996:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26238976, bytes: 4096 00000080:00200000:0.0:1789760773.647009:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.647019:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.647868:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26238976, 26243072) 0 0 00000080:00200000:0.0:1789760773.647881:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.647904:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6406, 6406] 00020000:00200000:0.0:1789760773.647914:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26238976 chunk: [26238976, 26243072) 26243072, 4096 00020000:00200000:0.0:1789760773.647918:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.647944:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6406, 6406] 00000008:00010000:0.0:1789760773.647964:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.647970:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760773.647985:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26238976, 26243072) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.648000:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26238976=0x1906000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.648071:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.648074:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26243072 00000020:00010000:0.0:1789760773.648083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.648086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.648089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.648092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.648096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.648114:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.648122:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.648147:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760773.648700:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 26238976<>26243072 00000100:00100000:0.0:1789760773.648753:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760773.648777:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495902592:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760773.689750:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973723 gen 2 00000008:00010000:1.0:1789760773.689803:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760773.689812:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.689823:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.689990:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760773.690032:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495902592:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760773.690046:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6406, 6406] 00000080:00200000:2.0:1789760773.690063:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.690068:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.690083:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26243072 i_blocks 53257 00000020:00200000:2.0:1789760773.690086:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.690094:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.690129:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26243072, count: 4096 00000080:00200000:2.0:1789760773.690136:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26243072, bytes: 4096 00000080:00200000:2.0:1789760773.690147:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.690156:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.690158:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26243072, 26247168) 0 0 00000080:00200000:2.0:1789760773.690163:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.690345:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6407, 6407] 00020000:00200000:2.0:1789760773.690353:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26243072 chunk: [26243072, 26247168) 26247168, 4096 00020000:00200000:2.0:1789760773.690358:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.690387:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6407, 6407] 00000008:00010000:2.0:1789760773.690405:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.690410:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760773.690422:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26243072, 26247168) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.690436:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26243072=0x1907000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.690485:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.690487:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26247168 00000020:00010000:2.0:1789760773.690590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.690596:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.690599:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.690603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.690606:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.690614:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.690621:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.690639:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760773.690919:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26243072<>26247168 00000100:00100000:3.0:1789760773.691011:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495902720:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760773.726085:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973724 gen 2 00000008:00010000:3.0:1789760773.726135:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760773.726144:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.726154:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.726206:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000080:00200000:2.0:1789760773.726218:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6407, 6407] 00000080:00200000:2.0:1789760773.726240:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760773.726240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495902720:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760773.726251:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.726268:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26247168 i_blocks 53257 00000020:00200000:2.0:1789760773.726272:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.726283:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.726319:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26247168, count: 4096 00000080:00200000:2.0:1789760773.726327:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26247168, bytes: 4096 00000080:00200000:2.0:1789760773.726336:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.726344:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.726346:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26247168, 26251264) 0 0 00000080:00200000:2.0:1789760773.726350:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.726364:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6408, 6408] 00020000:00200000:2.0:1789760773.726385:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26247168 chunk: [26247168, 26251264) 26251264, 4096 00020000:00200000:2.0:1789760773.726390:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.726412:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6408, 6408] 00000008:00010000:2.0:1789760773.726430:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.726435:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760773.726447:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26247168, 26251264) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.726461:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26247168=0x1908000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.726526:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.726529:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26251264 00000020:00010000:2.0:1789760773.726539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.726543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.726546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.726550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.726553:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.726560:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.726568:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.726584:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760773.726714:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 26247168<>26251264 00000100:00100000:3.0:1789760773.727409:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.727436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495902848:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760773.769831:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973725 gen 2 00000008:00010000:3.0:1789760773.769887:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760773.769896:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.769908:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.769938:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760773.769973:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495902848:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760773.770096:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6408, 6408] 00000080:00200000:2.0:1789760773.770112:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.770119:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.770134:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26251264 i_blocks 53257 00000020:00200000:2.0:1789760773.770138:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.770148:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.770215:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26251264, count: 4096 00000080:00200000:2.0:1789760773.770223:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26251264, bytes: 4096 00000080:00200000:2.0:1789760773.770233:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.770242:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.770245:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26251264, 26255360) 0 0 00000080:00200000:2.0:1789760773.770250:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.770267:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6409, 6409] 00020000:00200000:2.0:1789760773.770275:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26251264 chunk: [26251264, 26255360) 26255360, 4096 00020000:00200000:2.0:1789760773.770280:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.770301:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6409, 6409] 00000008:00010000:2.0:1789760773.770320:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.770326:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760773.770337:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26251264, 26255360) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.770351:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26251264=0x1909000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.770412:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.770415:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26255360 00000020:00010000:2.0:1789760773.770424:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.770428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.770431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.770435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.770438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.770445:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.770453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.770469:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760773.770589:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 26251264<>26255360 00000100:00100000:3.0:1789760773.770784:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495902976:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760773.822895:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760773.822912:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.822926:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.822954:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760773.822987:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495902976:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760773.823233:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6409, 6409] 00000080:00200000:2.0:1789760773.823251:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.823257:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.823275:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26255360 i_blocks 53257 00000020:00200000:2.0:1789760773.823279:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.823288:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.823320:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26255360, count: 4096 00000080:00200000:2.0:1789760773.823328:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26255360, bytes: 4096 00000080:00200000:2.0:1789760773.823340:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.823349:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.823351:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26255360, 26259456) 0 0 00000080:00200000:2.0:1789760773.823356:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.823382:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6410, 6410] 00020000:00200000:2.0:1789760773.823390:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26255360 chunk: [26255360, 26259456) 26259456, 4096 00020000:00200000:2.0:1789760773.823396:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.823416:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6410, 6410] 00000008:00010000:2.0:1789760773.823435:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.823442:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760773.823453:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26255360, 26259456) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.823466:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26255360=0x190a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.823519:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.823523:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26259456 00000020:00010000:2.0:1789760773.823532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.823536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.823539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.823543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.823546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.823553:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.823560:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.823577:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760773.823702:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 26255360<>26259456 00000100:00100000:3.0:1789760773.823823:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495903104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760773.883119:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973727 gen 2 00000008:00010000:3.0:1789760773.883204:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760773.883212:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.883222:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.883332:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760773.883378:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495903104:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760773.883676:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6410, 6410] 00000080:00200000:2.0:1789760773.883693:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.883699:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.883716:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26259456 i_blocks 53257 00000020:00200000:2.0:1789760773.883720:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.883731:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.883830:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26259456, count: 4096 00000080:00200000:2.0:1789760773.883838:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26259456, bytes: 4096 00000080:00200000:2.0:1789760773.883853:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.883862:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.883864:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26259456, 26263552) 0 0 00000080:00200000:2.0:1789760773.883868:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.883885:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6411, 6411] 00020000:00200000:2.0:1789760773.883894:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26259456 chunk: [26259456, 26263552) 26263552, 4096 00020000:00200000:2.0:1789760773.883899:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.883920:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6411, 6411] 00000008:00010000:2.0:1789760773.883937:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.883943:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760773.883955:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26259456, 26263552) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.883970:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26259456=0x190b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.884025:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.884028:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26263552 00000020:00010000:2.0:1789760773.884038:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.884042:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.884045:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.884049:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.884052:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.884059:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.884066:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.884084:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760773.884591:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26259456<>26263552 00000100:00100000:3.0:1789760773.884649:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495903232:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760773.937250:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973728 gen 2 00000008:00010000:3.0:1789760773.937301:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760773.937310:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.937321:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.937350:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760773.937389:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495903232:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760773.937506:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6411, 6411] 00000080:00200000:1.0:1789760773.937522:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760773.937529:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760773.937545:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26263552 i_blocks 53257 00000020:00200000:1.0:1789760773.937549:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760773.937560:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760773.937593:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26263552, count: 4096 00000080:00200000:1.0:1789760773.937601:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26263552, bytes: 4096 00000080:00200000:1.0:1789760773.937610:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760773.937619:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760773.937621:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26263552, 26267648) 0 0 00000080:00200000:1.0:1789760773.937626:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760773.937642:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6412, 6412] 00020000:00200000:1.0:1789760773.937651:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26263552 chunk: [26263552, 26267648) 26267648, 4096 00020000:00200000:1.0:1789760773.937656:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760773.937831:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6412, 6412] 00000008:00010000:1.0:1789760773.937851:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760773.937857:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760773.937870:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26263552, 26267648) append 0 append lockpos 0 00000080:00200000:1.0:1789760773.937884:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26263552=0x190c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760773.937945:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760773.937949:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26267648 00000020:00010000:1.0:1789760773.937957:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760773.937960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760773.937964:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760773.937967:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760773.937970:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760773.937976:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760773.937984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760773.937999:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760773.938124:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26263552<>26267648 00000100:00100000:1.0:1789760773.938205:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760773.938224:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495903360:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760773.991671:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973729 gen 2 00000008:00010000:1.0:1789760773.991719:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760773.991727:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.991740:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.991772:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760773.991802:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495903360:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760773.991967:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6412, 6412] 00000080:00200000:0.0:1789760773.991984:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.991991:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.992007:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26267648 i_blocks 53257 00000020:00200000:0.0:1789760773.992010:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.992021:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.992054:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26267648, count: 4096 00000080:00200000:0.0:1789760773.992060:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26267648, bytes: 4096 00000080:00200000:0.0:1789760773.992076:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.992085:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.992088:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26267648, 26271744) 0 0 00000080:00200000:0.0:1789760773.992093:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.992108:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6413, 6413] 00020000:00200000:0.0:1789760773.992118:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26267648 chunk: [26267648, 26271744) 26271744, 4096 00020000:00200000:0.0:1789760773.992123:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.992144:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6413, 6413] 00000008:00010000:0.0:1789760773.992162:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.992199:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760773.992212:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26267648, 26271744) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.992228:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26267648=0x190d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.992283:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.992286:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26271744 00000020:00010000:0.0:1789760773.992295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.992298:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.992301:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0098/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.992305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.992308:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.992315:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.992323:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.992339:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760773.992480:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 26267648<>26271744 00000100:00100000:0.0:1789760773.992523:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760773.992540:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495903488:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760774.021492:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973730 gen 2 00000008:00010000:1.0:1789760774.021538:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760774.021546:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.021556:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.021624:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760774.021662:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495903488:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760774.021978:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6413, 6413] 00000080:00200000:0.0:1789760774.021995:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.022001:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.022108:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26271744 i_blocks 53257 00000020:00200000:1.0:1789760774.022113:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.022122:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.022158:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26271744, count: 4096 00000080:00200000:1.0:1789760774.022165:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26271744, bytes: 4096 00000080:00200000:1.0:1789760774.022212:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.022220:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.022223:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26271744, 26275840) 0 0 00000080:00200000:1.0:1789760774.022227:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.022244:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6414, 6414] 00020000:00200000:1.0:1789760774.022252:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26271744 chunk: [26271744, 26275840) 26275840, 4096 00020000:00200000:1.0:1789760774.022257:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.022277:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6414, 6414] 00000008:00010000:1.0:1789760774.022295:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.022300:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.022312:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26271744, 26275840) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.022328:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26271744=0x190e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.022498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.022501:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26275840 00000020:00010000:1.0:1789760774.022511:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.022514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.022517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.022520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.022523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.022532:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.022538:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.022554:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760774.022679:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26271744<>26275840 00000100:00100000:0.0:1789760774.022955:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495903616:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.027407:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495903872:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760774.027647:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1789760774.027654:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495903744:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760774.027700:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2357:1876699495904000:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760774.035291:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495903744:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760774.035319:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2357:1876699495904000:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760774.035331:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495903872:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760774.078112:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973731 gen 2 00000008:00010000:0.0:1789760774.078160:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760774.078197:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.078208:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.078228:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760774.078256:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495903616:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.078424:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6414, 6414] 00000080:00200000:1.0:1789760774.078440:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.078446:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.078461:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26275840 i_blocks 53257 00000020:00200000:1.0:1789760774.078465:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.078473:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.078505:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26275840, count: 4096 00000080:00200000:1.0:1789760774.078512:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26275840, bytes: 4096 00000080:00200000:1.0:1789760774.078522:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.078530:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.078533:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26275840, 26279936) 0 0 00000080:00200000:1.0:1789760774.078537:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.078553:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6415, 6415] 00020000:00200000:1.0:1789760774.078562:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26275840 chunk: [26275840, 26279936) 26279936, 4096 00020000:00200000:1.0:1789760774.078567:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.078585:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6415, 6415] 00000008:00010000:1.0:1789760774.078602:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.078608:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.078619:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26275840, 26279936) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.078632:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26275840=0x190f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.078679:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.078694:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26279936 00000020:00010000:1.0:1789760774.078701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.078704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.078707:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.078711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.078714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.078720:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.078726:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.078741:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760774.078873:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 26275840<>26279936 00000100:00100000:1.0:1789760774.078909:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760774.078925:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495904128:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.121536:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973732 gen 2 00000008:00010000:0.0:1789760774.121708:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760774.121716:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.121727:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.121790:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760774.121820:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495904128:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.122027:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6415, 6415] 00000080:00200000:1.0:1789760774.122044:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.122051:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.122066:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26279936 i_blocks 53257 00000020:00200000:1.0:1789760774.122070:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.122143:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.122207:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26279936, count: 4096 00000080:00200000:1.0:1789760774.122214:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26279936, bytes: 4096 00000080:00200000:1.0:1789760774.122224:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.122232:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.122234:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26279936, 26284032) 0 0 00000080:00200000:1.0:1789760774.122239:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.122253:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6416, 6416] 00020000:00200000:1.0:1789760774.122262:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26279936 chunk: [26279936, 26284032) 26284032, 4096 00020000:00200000:1.0:1789760774.122267:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.122285:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6416, 6416] 00000008:00010000:1.0:1789760774.122302:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.122308:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.122318:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26279936, 26284032) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.122331:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26279936=0x1910000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.122394:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.122397:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26284032 00000020:00010000:1.0:1789760774.122405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.122408:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.122411:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.122415:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.122417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.122424:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.122430:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.122446:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760774.122569:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 26279936<>26284032 00000100:00100000:0.0:1789760774.122696:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495904256:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.163779:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973733 gen 2 00000008:00010000:0.0:1789760774.163839:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760774.163850:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.163863:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.163958:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760774.164000:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495904256:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.164955:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6416, 6416] 00000080:00200000:1.0:1789760774.164974:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.164979:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.164994:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26284032 i_blocks 53257 00000020:00200000:1.0:1789760774.164998:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.165008:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.165039:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26284032, count: 4096 00000080:00200000:1.0:1789760774.165046:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26284032, bytes: 4096 00000080:00200000:1.0:1789760774.165055:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.165063:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.165065:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26284032, 26288128) 0 0 00000080:00200000:1.0:1789760774.165068:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.165083:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6417, 6417] 00020000:00200000:1.0:1789760774.165091:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26284032 chunk: [26284032, 26288128) 26288128, 4096 00020000:00200000:1.0:1789760774.165094:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.165112:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6417, 6417] 00000008:00010000:1.0:1789760774.165128:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.165133:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.165144:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26284032, 26288128) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.165155:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26284032=0x1911000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.165222:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.165224:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26288128 00000020:00010000:1.0:1789760774.165231:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.165234:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.165236:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.165239:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.165241:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.165246:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.165251:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.165269:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760774.165476:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 26284032<>26288128 00000100:00100000:0.0:1789760774.165575:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495904384:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760774.195487:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760774.195503:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.195515:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.195542:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760774.195573:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495904384:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.196218:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6417, 6417] 00000080:00200000:1.0:1789760774.196238:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.196245:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.196261:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26288128 i_blocks 53257 00000020:00200000:1.0:1789760774.196264:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.196274:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.196309:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26288128, count: 4096 00000080:00200000:1.0:1789760774.196317:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26288128, bytes: 4096 00000080:00200000:1.0:1789760774.196332:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.196342:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.196344:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26288128, 26292224) 0 0 00000080:00200000:1.0:1789760774.196426:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.196446:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6418, 6418] 00020000:00200000:1.0:1789760774.196455:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26288128 chunk: [26288128, 26292224) 26292224, 4096 00020000:00200000:1.0:1789760774.196460:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.196481:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6418, 6418] 00000008:00010000:1.0:1789760774.196501:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.196507:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.196519:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26288128, 26292224) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.196535:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26288128=0x1912000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.196607:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.196610:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26292224 00000020:00010000:1.0:1789760774.196620:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.196624:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.196628:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.196631:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.196634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.196643:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.196650:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.196669:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760774.196798:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 26288128<>26292224 00000100:00100000:0.0:1789760774.197259:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.197283:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495904512:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.275689:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973734 gen 2 00000100:00100000:0.0:1789760774.275728:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504c000 x1876699495904512/t4294973735(4294973735) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760790 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760774.275777:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760774.275784:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.275795:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.275965:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760774.275993:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495904512:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.276321:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6418, 6418] 00000080:00200000:1.0:1789760774.276345:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.276351:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.276375:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26292224 i_blocks 53257 00000020:00200000:1.0:1789760774.276379:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.276388:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.276621:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26292224, count: 4096 00000080:00200000:1.0:1789760774.276631:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26292224, bytes: 4096 00000080:00200000:1.0:1789760774.276643:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.276651:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.276654:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26292224, 26296320) 0 0 00000080:00200000:1.0:1789760774.276659:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.276674:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6419, 6419] 00020000:00200000:1.0:1789760774.276683:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26292224 chunk: [26292224, 26296320) 26296320, 4096 00020000:00200000:1.0:1789760774.276687:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.276704:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6419, 6419] 00000008:00010000:1.0:1789760774.276732:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.276738:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.276749:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26292224, 26296320) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.276764:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26292224=0x1913000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.276818:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.276820:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26296320 00000020:00010000:1.0:1789760774.276828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.276832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.276835:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.276838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.276840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.276848:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.276854:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.276871:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760774.276999:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 26292224<>26296320 00000100:00100000:1.0:1789760774.277673:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495904640:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.317263:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973736 gen 2 00000008:00010000:0.0:1789760774.317336:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760774.317413:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.317424:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.317445:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760774.317476:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495904640:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.317576:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6419, 6419] 00000080:00200000:1.0:1789760774.318913:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.318924:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.318946:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26296320 i_blocks 53257 00000020:00200000:1.0:1789760774.318951:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.318965:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.319133:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26296320, count: 4096 00000080:00200000:1.0:1789760774.319144:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26296320, bytes: 4096 00000080:00200000:1.0:1789760774.319160:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.319190:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.319196:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26296320, 26300416) 0 0 00000080:00200000:1.0:1789760774.319204:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.319234:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6420, 6420] 00020000:00200000:1.0:1789760774.319248:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26296320 chunk: [26296320, 26300416) 26300416, 4096 00020000:00200000:1.0:1789760774.319256:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.319292:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6420, 6420] 00000008:00010000:1.0:1789760774.319319:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.319328:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.319410:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26296320, 26300416) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.319435:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26296320=0x1914000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.319912:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.319917:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26300416 00000020:00010000:1.0:1789760774.319927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.319931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.319934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.319938:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.319940:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.319948:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.319955:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.319985:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760774.321224:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 26296320<>26300416 00000100:00100000:1.0:1789760774.321502:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760774.321542:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495904768:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.376948:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973737 gen 2 00000008:00010000:0.0:1789760774.376997:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760774.377006:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.377017:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.377084:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760774.377113:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495904768:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760774.383235:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6420, 6420] 00000080:00200000:3.0:1789760774.383264:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.383271:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.383291:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26300416 i_blocks 53257 00000020:00200000:3.0:1789760774.383296:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.383313:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.383403:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26300416, count: 4096 00000080:00200000:3.0:1789760774.383412:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26300416, bytes: 4096 00000080:00200000:3.0:1789760774.383427:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.383436:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760774.383439:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26300416, 26304512) 0 0 00000080:00200000:3.0:1789760774.383444:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.383461:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6421, 6421] 00020000:00200000:3.0:1789760774.383469:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26300416 chunk: [26300416, 26304512) 26304512, 4096 00020000:00200000:3.0:1789760774.383474:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.383497:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6421, 6421] 00000008:00010000:3.0:1789760774.383518:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.383524:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760774.383536:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26300416, 26304512) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.383552:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26300416=0x1915000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.383624:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.383627:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26304512 00000020:00010000:3.0:1789760774.383636:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.383640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.383643:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.383647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.383650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.383659:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.383666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.383708:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760774.383845:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 26300416<>26304512 00000100:00100000:3.0:1789760774.383902:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760774.383928:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495904896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760774.424417:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973738 gen 2 00000008:00010000:3.0:1789760774.424464:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:3.0:1789760774.424472:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.424484:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.424498:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:3.0:1789760774.424528:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495904896:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760774.424718:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6421, 6421] 00000080:00200000:0.0:1789760774.424734:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.424741:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.424756:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26304512 i_blocks 53257 00000020:00200000:0.0:1789760774.424759:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.424768:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.424801:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26304512, count: 4096 00000080:00200000:0.0:1789760774.424808:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26304512, bytes: 4096 00000080:00200000:0.0:1789760774.424819:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.424828:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.424830:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26304512, 26308608) 0 0 00000080:00200000:0.0:1789760774.424835:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.424851:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6422, 6422] 00020000:00200000:0.0:1789760774.424859:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26304512 chunk: [26304512, 26308608) 26308608, 4096 00020000:00200000:0.0:1789760774.424863:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.424881:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6422, 6422] 00000008:00010000:0.0:1789760774.424897:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.424903:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760774.424915:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26304512, 26308608) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.424928:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26304512=0x1916000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.424984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.424987:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26308608 00000020:00010000:0.0:1789760774.424995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.424999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.425003:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.425007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.425010:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.425017:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.425024:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.425041:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760774.425873:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26304512<>26308608 00000100:00100000:0.0:1789760774.425922:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760774.425942:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905024:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.467809:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973739 gen 2 00000008:00010000:0.0:1789760774.467868:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760774.467876:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.467888:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.467921:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760774.467950:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905024:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760774.468086:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6422, 6422] 00000080:00200000:3.0:1789760774.468104:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.468111:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.468125:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26308608 i_blocks 53257 00000020:00200000:3.0:1789760774.468129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.468139:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.468204:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26308608, count: 4096 00000080:00200000:3.0:1789760774.468212:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26308608, bytes: 4096 00000080:00200000:3.0:1789760774.468227:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.468236:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760774.468239:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26308608, 26312704) 0 0 00000080:00200000:3.0:1789760774.468243:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.468260:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6423, 6423] 00020000:00200000:3.0:1789760774.468269:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26308608 chunk: [26308608, 26312704) 26312704, 4096 00020000:00200000:3.0:1789760774.468274:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.468294:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6423, 6423] 00000008:00010000:3.0:1789760774.468311:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.468316:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760774.468328:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26308608, 26312704) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.468343:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26308608=0x1917000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.468421:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.468424:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26312704 00000020:00010000:3.0:1789760774.468432:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.468436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.468439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.468442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.468446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.468453:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.468460:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.468477:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760774.468599:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 26308608<>26312704 00000100:00100000:2.0:1789760774.468731:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760774.468749:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495905152:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760774.504053:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973740 gen 2 00000008:00010000:2.0:1789760774.504103:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760774.504112:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760774.504124:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760774.504142:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000080:00200000:3.0:1789760774.504197:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6423, 6423] 00000080:00200000:3.0:1789760774.504218:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1789760774.504219:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495905152:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760774.504231:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.504248:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26312704 i_blocks 53257 00000020:00200000:3.0:1789760774.504252:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.504262:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.504294:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26312704, count: 4096 00000080:00200000:3.0:1789760774.504301:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26312704, bytes: 4096 00000080:00200000:3.0:1789760774.504309:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.504317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760774.504320:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26312704, 26316800) 0 0 00000080:00200000:3.0:1789760774.504324:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.504342:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6424, 6424] 00020000:00200000:3.0:1789760774.504350:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26312704 chunk: [26312704, 26316800) 26316800, 4096 00020000:00200000:3.0:1789760774.504354:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.504482:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6424, 6424] 00000008:00010000:3.0:1789760774.504501:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.504508:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760774.504520:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26312704, 26316800) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.504534:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26312704=0x1918000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.504586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.504589:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26316800 00000020:00010000:3.0:1789760774.504597:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.504600:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.504603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.504607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.504611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.504619:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.504625:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.504641:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760774.504980:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 26312704<>26316800 00000100:00100000:2.0:1789760774.505086:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495905280:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760774.545670:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973741 gen 2 00000008:00010000:3.0:1789760774.545721:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:3.0:1789760774.545729:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.545740:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.545767:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:3.0:1789760774.545800:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495905280:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.545900:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6424, 6424] 00000080:00200000:1.0:1789760774.545916:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.545922:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.545936:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26316800 i_blocks 53257 00000020:00200000:1.0:1789760774.545939:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.545948:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.545982:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26316800, count: 4096 00000080:00200000:1.0:1789760774.545989:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26316800, bytes: 4096 00000080:00200000:1.0:1789760774.546002:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.546011:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.546013:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26316800, 26320896) 0 0 00000080:00200000:1.0:1789760774.546018:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.546035:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6425, 6425] 00020000:00200000:1.0:1789760774.546044:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26316800 chunk: [26316800, 26320896) 26320896, 4096 00020000:00200000:1.0:1789760774.546049:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.546069:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6425, 6425] 00000008:00010000:1.0:1789760774.546088:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.546094:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.546105:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26316800, 26320896) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.546118:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26316800=0x1919000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.546193:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.546196:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26320896 00000020:00010000:1.0:1789760774.546205:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.546208:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.546211:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.546215:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.546217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.546225:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.546231:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.546249:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760774.546596:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 26316800<>26320896 00000100:00100000:1.0:1789760774.546633:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760774.546652:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495905408:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.584429:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973742 gen 2 00000008:00010000:0.0:1789760774.584480:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760774.584488:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.584499:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.584522:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760774.584553:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495905408:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.584607:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6425, 6425] 00000080:00200000:1.0:1789760774.584625:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.584632:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.584648:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26320896 i_blocks 53257 00000020:00200000:1.0:1789760774.584650:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.584668:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.584702:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26320896, count: 4096 00000080:00200000:1.0:1789760774.584710:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26320896, bytes: 4096 00000080:00200000:1.0:1789760774.584719:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.584728:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.584731:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26320896, 26324992) 0 0 00000080:00200000:1.0:1789760774.584736:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.584754:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6426, 6426] 00020000:00200000:1.0:1789760774.584763:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26320896 chunk: [26320896, 26324992) 26324992, 4096 00020000:00200000:1.0:1789760774.584768:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.584789:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6426, 6426] 00000008:00010000:1.0:1789760774.584806:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.584812:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.584823:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26320896, 26324992) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.584845:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26320896=0x191a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.584914:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.584917:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26324992 00000020:00010000:1.0:1789760774.584924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.584927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.584930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.584932:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.584934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.584940:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.584945:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.584958:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760774.585098:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 26320896<>26324992 00000100:00100000:1.0:1789760774.585146:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760774.585160:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905536:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760774.604422:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760774.634014:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973743 gen 2 00000008:00010000:0.0:1789760774.634066:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760774.634074:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.634085:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.634108:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760774.634138:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905536:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.634301:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6426, 6426] 00000080:00200000:1.0:1789760774.634319:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.634329:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.634345:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26324992 i_blocks 53257 00000020:00200000:1.0:1789760774.634349:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.634368:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.634402:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26324992, count: 4096 00000080:00200000:1.0:1789760774.634410:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26324992, bytes: 4096 00000080:00200000:1.0:1789760774.634420:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.634429:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.634431:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26324992, 26329088) 0 0 00000080:00200000:1.0:1789760774.634436:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.634453:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6427, 6427] 00020000:00200000:1.0:1789760774.634461:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26324992 chunk: [26324992, 26329088) 26329088, 4096 00020000:00200000:1.0:1789760774.634466:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.634487:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6427, 6427] 00000008:00010000:1.0:1789760774.634504:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.634511:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.634522:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26324992, 26329088) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.634659:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26324992=0x191b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.634788:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.634792:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26329088 00000020:00010000:1.0:1789760774.634800:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.634804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.634807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.634810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.634813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.634821:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.634828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.634844:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760774.634970:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 26324992<>26329088 00000100:00100000:0.0:1789760774.635133:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905664:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.683978:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973744 gen 2 00000008:00010000:0.0:1789760774.684029:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760774.684038:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.684050:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.684063:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760774.684093:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905664:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.684438:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6427, 6427] 00000080:00200000:1.0:1789760774.684458:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.684465:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.684482:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26329088 i_blocks 53257 00000020:00200000:1.0:1789760774.684486:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.684496:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.684532:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26329088, count: 4096 00000080:00200000:1.0:1789760774.684539:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26329088, bytes: 4096 00000080:00200000:1.0:1789760774.684552:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.684562:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.684564:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26329088, 26333184) 0 0 00000080:00200000:1.0:1789760774.684569:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.684586:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6428, 6428] 00020000:00200000:1.0:1789760774.684594:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26329088 chunk: [26329088, 26333184) 26333184, 4096 00020000:00200000:1.0:1789760774.684598:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.684619:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6428, 6428] 00000008:00010000:1.0:1789760774.684637:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.684643:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.684655:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26329088, 26333184) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.684669:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26329088=0x191c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.684728:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.684731:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26333184 00000020:00010000:1.0:1789760774.684740:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.684744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.684747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.684750:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.684753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.684760:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.684768:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.684784:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760774.685078:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 26329088<>26333184 00000100:00100000:0.0:1789760774.685279:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760774.685300:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905792:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.725647:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973745 gen 2 00000008:00010000:0.0:1789760774.725699:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760774.725708:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.725720:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.725785:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760774.725817:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905792:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.725899:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6428, 6428] 00000080:00200000:1.0:1789760774.725917:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.725924:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.725941:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26333184 i_blocks 53257 00000020:00200000:1.0:1789760774.725946:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.725956:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.725992:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26333184, count: 4096 00000080:00200000:1.0:1789760774.726010:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26333184, bytes: 4096 00000080:00200000:1.0:1789760774.726020:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.726029:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.726135:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26333184, 26337280) 0 0 00000080:00200000:1.0:1789760774.726142:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.726160:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6429, 6429] 00020000:00200000:1.0:1789760774.726184:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26333184 chunk: [26333184, 26337280) 26337280, 4096 00020000:00200000:1.0:1789760774.726188:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.726207:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6429, 6429] 00000008:00010000:1.0:1789760774.726222:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.726227:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.726238:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26333184, 26337280) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.726252:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26333184=0x191d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.726309:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.726313:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26337280 00000020:00010000:1.0:1789760774.726435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.726441:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.726444:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.726447:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.726450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.726460:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.726468:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.726485:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760774.726615:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 26333184<>26337280 00000100:00100000:0.0:1789760774.726684:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905920:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760774.754030:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760774.754046:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.754058:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.754077:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760774.754110:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495905920:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.754196:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6429, 6429] 00000080:00200000:1.0:1789760774.754211:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.754217:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.754231:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26337280 i_blocks 53257 00000020:00200000:1.0:1789760774.754233:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.754241:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.754273:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26337280, count: 4096 00000080:00200000:1.0:1789760774.754279:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26337280, bytes: 4096 00000080:00200000:1.0:1789760774.754286:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.754294:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.754295:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26337280, 26341376) 0 0 00000080:00200000:1.0:1789760774.754299:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.754311:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6430, 6430] 00020000:00200000:1.0:1789760774.754318:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26337280 chunk: [26337280, 26341376) 26341376, 4096 00020000:00200000:1.0:1789760774.754322:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.754400:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6430, 6430] 00000008:00010000:1.0:1789760774.754420:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.754427:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.754438:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26337280, 26341376) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.754453:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26337280=0x191e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.754509:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.754512:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26341376 00000020:00010000:1.0:1789760774.754520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.754524:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.754527:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.754531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.754534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.754541:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.754548:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.754565:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760774.754706:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 26337280<>26341376 00000100:00100000:0.0:1789760774.754808:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906048:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.783101:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973746 gen 2 00000100:00100000:0.0:1789760774.783144:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d500 x1876699495906048/t4294973747(4294973747) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760790 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760774.783311:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760774.783319:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.783330:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.783475:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760774.783503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906048:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.783574:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6430, 6430] 00000080:00200000:1.0:1789760774.783593:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.783600:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.783616:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26341376 i_blocks 53257 00000020:00200000:1.0:1789760774.783620:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.783630:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.783679:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26341376, count: 4096 00000080:00200000:1.0:1789760774.783686:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26341376, bytes: 4096 00000080:00200000:1.0:1789760774.783697:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.783706:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.783709:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26341376, 26345472) 0 0 00000080:00200000:1.0:1789760774.783713:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.783731:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6431, 6431] 00020000:00200000:1.0:1789760774.783739:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26341376 chunk: [26341376, 26345472) 26345472, 4096 00020000:00200000:1.0:1789760774.783745:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.783765:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6431, 6431] 00000008:00010000:1.0:1789760774.783784:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.783789:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.783799:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26341376, 26345472) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.783814:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26341376=0x191f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.783872:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.783875:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26345472 00000020:00010000:1.0:1789760774.783883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.783887:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.783890:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.783893:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.783896:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.783904:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.783911:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.783926:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760774.784040:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 26341376<>26345472 00000100:00100000:0.0:1789760774.784093:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.784109:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906176:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.825253:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973748 gen 2 00000008:00010000:0.0:1789760774.825318:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760774.825327:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.825337:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.825367:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760774.825396:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906176:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.825543:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6431, 6431] 00000080:00200000:1.0:1789760774.825561:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.825568:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.825585:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26345472 i_blocks 53257 00000020:00200000:1.0:1789760774.825589:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.825599:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.825633:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26345472, count: 4096 00000080:00200000:1.0:1789760774.825640:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26345472, bytes: 4096 00000080:00200000:1.0:1789760774.825651:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.825661:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.825663:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26345472, 26349568) 0 0 00000080:00200000:1.0:1789760774.825668:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.825684:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6432, 6432] 00020000:00200000:1.0:1789760774.825693:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26345472 chunk: [26345472, 26349568) 26349568, 4096 00020000:00200000:1.0:1789760774.825697:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.825717:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6432, 6432] 00000008:00010000:1.0:1789760774.825734:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.825740:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.825751:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26345472, 26349568) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.825766:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26345472=0x1920000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.825821:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.825824:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26349568 00000020:00010000:1.0:1789760774.825832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.825836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.825839:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.825842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.825846:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.825853:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.825860:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.825877:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760774.826002:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 26345472<>26349568 00000100:00100000:1.0:1789760774.827681:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760774.827712:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495906304:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760774.887930:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760774.887946:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.887960:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.887992:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760774.888038:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495906304:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.888421:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6432, 6432] 00000080:00200000:1.0:1789760774.888438:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.888445:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.888461:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26349568 i_blocks 53257 00000020:00200000:1.0:1789760774.888464:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.888475:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.888511:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26349568, count: 4096 00000080:00200000:1.0:1789760774.888520:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26349568, bytes: 4096 00000080:00200000:1.0:1789760774.888535:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.888545:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.888547:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26349568, 26353664) 0 0 00000080:00200000:1.0:1789760774.888553:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.888570:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6433, 6433] 00020000:00200000:1.0:1789760774.888579:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26349568 chunk: [26349568, 26353664) 26353664, 4096 00020000:00200000:1.0:1789760774.888584:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.888606:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6433, 6433] 00000008:00010000:1.0:1789760774.888626:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.888633:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.888646:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26349568, 26353664) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.888661:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26349568=0x1921000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.888721:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.888724:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26353664 00000020:00010000:1.0:1789760774.888733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.888736:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.888740:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.888744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.888747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.888754:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.888762:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.888780:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760774.888915:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 26349568<>26353664 00000100:00100000:0.0:1789760774.889816:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.889844:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906432:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760774.919635:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973750 gen 2 00000008:00010000:0.0:1789760774.919703:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760774.919711:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.919719:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.919735:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760774.919764:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906432:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.920082:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6433, 6433] 00000080:00200000:1.0:1789760774.920101:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.920108:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.920124:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26353664 i_blocks 53257 00000020:00200000:1.0:1789760774.920128:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.920138:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.920207:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26353664, count: 4096 00000080:00200000:1.0:1789760774.920216:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26353664, bytes: 4096 00000080:00200000:1.0:1789760774.920225:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.920234:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.920236:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26353664, 26357760) 0 0 00000080:00200000:1.0:1789760774.920240:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.920257:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6434, 6434] 00020000:00200000:1.0:1789760774.920265:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26353664 chunk: [26353664, 26357760) 26357760, 4096 00020000:00200000:1.0:1789760774.920270:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.920291:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6434, 6434] 00000008:00010000:1.0:1789760774.920309:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.920315:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.920327:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26353664, 26357760) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.920341:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26353664=0x1922000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.920408:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.920411:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26357760 00000020:00010000:1.0:1789760774.920419:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.920423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.920426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.920430:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.920433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.920441:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.920448:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.920476:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760774.920605:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 26353664<>26357760 00000100:00100000:0.0:1789760774.920790:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760774.920810:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495906560:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760774.957148:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760774.957165:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.957208:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.957230:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760774.957263:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495906560:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760774.957267:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6434, 6434] 00000080:00200000:1.0:1789760774.957287:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.957352:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.957376:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26357760 i_blocks 53257 00000020:00200000:1.0:1789760774.957494:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.957504:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.957539:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26357760, count: 4096 00000080:00200000:1.0:1789760774.957547:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26357760, bytes: 4096 00000080:00200000:1.0:1789760774.957557:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.957565:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.957567:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26357760, 26361856) 0 0 00000080:00200000:1.0:1789760774.957930:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.957948:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6435, 6435] 00020000:00200000:1.0:1789760774.957956:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26357760 chunk: [26357760, 26361856) 26361856, 4096 00020000:00200000:1.0:1789760774.957961:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.957981:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6435, 6435] 00000008:00010000:1.0:1789760774.958000:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.958006:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760774.958017:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26357760, 26361856) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.958032:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26357760=0x1923000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.958649:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.958654:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26361856 00000020:00010000:1.0:1789760774.958663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.958666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.958670:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.958674:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.958677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.958687:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.958695:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.958721:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760774.958852:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 26357760<>26361856 00000100:00100000:0.0:1789760774.958961:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.958989:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906688:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.007272:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973752 gen 2 00000008:00010000:0.0:1789760775.007347:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760775.007365:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.007376:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.007395:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760775.007425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906688:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.007526:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6435, 6435] 00000080:00200000:1.0:1789760775.007542:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.007549:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.007566:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26361856 i_blocks 53257 00000020:00200000:1.0:1789760775.007570:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.007579:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.007616:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26361856, count: 4096 00000080:00200000:1.0:1789760775.007624:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26361856, bytes: 4096 00000080:00200000:1.0:1789760775.007634:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.007644:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.007647:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26361856, 26365952) 0 0 00000080:00200000:1.0:1789760775.007651:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.007668:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6436, 6436] 00020000:00200000:1.0:1789760775.007738:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26361856 chunk: [26361856, 26365952) 26365952, 4096 00020000:00200000:1.0:1789760775.007744:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.007767:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6436, 6436] 00000008:00010000:1.0:1789760775.007787:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.007793:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.007804:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26361856, 26365952) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.007823:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26361856=0x1924000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.007894:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.007898:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26365952 00000020:00010000:1.0:1789760775.007906:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.008029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.008033:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.008037:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.008040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.008050:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.008057:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.008075:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760775.008344:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 26361856<>26365952 00000100:00100000:0.0:1789760775.008414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906816:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.039527:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973753 gen 2 00000008:00010000:0.0:1789760775.039577:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760775.039586:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.039596:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.039670:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760775.039753:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495906816:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760775.039822:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6436, 6436] 00000080:00200000:3.0:1789760775.039835:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.039840:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.039854:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26365952 i_blocks 53257 00000020:00200000:3.0:1789760775.039858:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.039868:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.039902:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26365952, count: 4096 00000080:00200000:3.0:1789760775.039910:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26365952, bytes: 4096 00000080:00200000:3.0:1789760775.039920:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.039929:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760775.039932:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26365952, 26370048) 0 0 00000080:00200000:3.0:1789760775.039936:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.039953:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6437, 6437] 00020000:00200000:3.0:1789760775.039960:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26365952 chunk: [26365952, 26370048) 26370048, 4096 00020000:00200000:3.0:1789760775.039963:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.039981:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6437, 6437] 00000008:00010000:3.0:1789760775.039995:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.039999:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760775.040008:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26365952, 26370048) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.040021:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26365952=0x1925000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.040078:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.040081:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26370048 00000020:00010000:3.0:1789760775.040089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.040092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.040097:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.040100:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.040103:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.040110:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.040117:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.040133:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760775.040425:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 26365952<>26370048 00000100:00100000:2.0:1789760775.040483:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495906944:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760775.078275:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973754 gen 2 00000008:00010000:2.0:1789760775.078324:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760775.078333:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760775.078344:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760775.078378:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760775.078409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495906944:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760775.078577:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6437, 6437] 00000080:00200000:0.0:1789760775.078594:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.078600:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.078616:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26370048 i_blocks 53257 00000020:00200000:0.0:1789760775.078619:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.078639:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.078674:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26370048, count: 4096 00000080:00200000:0.0:1789760775.078680:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26370048, bytes: 4096 00000080:00200000:0.0:1789760775.078691:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.078700:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.078703:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26370048, 26374144) 0 0 00000080:00200000:0.0:1789760775.078707:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.078723:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6438, 6438] 00020000:00200000:0.0:1789760775.078731:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26370048 chunk: [26370048, 26374144) 26374144, 4096 00020000:00200000:0.0:1789760775.080062:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.080099:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6438, 6438] 00000008:00010000:0.0:1789760775.080127:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.080134:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760775.080148:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26370048, 26374144) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.080164:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26370048=0x1926000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.080251:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.080256:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26374144 00000020:00010000:0.0:1789760775.080264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.080268:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.080272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.080275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.080279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.080290:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.080916:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.080954:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760775.081103:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26370048<>26374144 00000100:00100000:1.0:1789760775.081214:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907072:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.110515:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973755 gen 2 00000008:00010000:0.0:1789760775.110566:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760775.110575:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.110585:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.110607:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760775.110638:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907072:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760775.110651:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6438, 6438] 00000080:00200000:2.0:1789760775.110671:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.110677:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.110692:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26374144 i_blocks 53257 00000020:00200000:2.0:1789760775.110696:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.110706:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.110739:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26374144, count: 4096 00000080:00200000:2.0:1789760775.110808:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26374144, bytes: 4096 00000080:00200000:2.0:1789760775.110819:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.110829:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.110832:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26374144, 26378240) 0 0 00000080:00200000:2.0:1789760775.110836:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.110854:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6439, 6439] 00020000:00200000:2.0:1789760775.110862:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26374144 chunk: [26374144, 26378240) 26378240, 4096 00020000:00200000:2.0:1789760775.110930:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.110950:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6439, 6439] 00000008:00010000:2.0:1789760775.110968:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.110973:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760775.110985:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26374144, 26378240) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.111063:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26374144=0x1927000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.111126:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.111129:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26378240 00000020:00010000:2.0:1789760775.111139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.111142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.111145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.111149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.111152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.111159:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.111200:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.111218:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760775.111415:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26374144<>26378240 00000100:00100000:2.0:1789760775.111461:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760775.111479:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495907200:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760775.154595:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973756 gen 2 00000008:00010000:2.0:1789760775.154647:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760775.154656:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760775.154669:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760775.154699:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760775.154729:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495907200:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760775.154940:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6439, 6439] 00000080:00200000:3.0:1789760775.154959:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.154966:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.154983:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26378240 i_blocks 53257 00000020:00200000:3.0:1789760775.154986:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.155007:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.155045:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26378240, count: 4096 00000080:00200000:3.0:1789760775.155053:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26378240, bytes: 4096 00000080:00200000:3.0:1789760775.155063:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.155071:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760775.155073:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26378240, 26382336) 0 0 00000080:00200000:3.0:1789760775.155078:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.155093:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6440, 6440] 00020000:00200000:3.0:1789760775.155101:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26378240 chunk: [26378240, 26382336) 26382336, 4096 00020000:00200000:3.0:1789760775.155105:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.155125:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6440, 6440] 00000008:00010000:3.0:1789760775.155142:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.155148:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760775.155160:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26378240, 26382336) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.155208:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26378240=0x1928000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.155268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.155271:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26382336 00000020:00010000:3.0:1789760775.155281:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.155285:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.155288:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.155291:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.155294:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.155302:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.155309:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.155326:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760775.155467:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 26378240<>26382336 00000100:00100000:3.0:1789760775.155512:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760775.155530:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495907328:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760775.186014:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:3.0:1789760775.186031:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.186042:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.186070:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:3.0:1789760775.186101:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495907328:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760775.186342:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6440, 6440] 00000080:00200000:0.0:1789760775.186369:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.186375:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.186392:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26382336 i_blocks 53257 00000020:00200000:0.0:1789760775.186396:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.186405:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.186440:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26382336, count: 4096 00000080:00200000:0.0:1789760775.186448:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26382336, bytes: 4096 00000080:00200000:0.0:1789760775.186458:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.186467:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.186470:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26382336, 26386432) 0 0 00000080:00200000:0.0:1789760775.186474:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.186492:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6441, 6441] 00020000:00200000:0.0:1789760775.186501:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26382336 chunk: [26382336, 26386432) 26386432, 4096 00020000:00200000:0.0:1789760775.186505:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.186528:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6441, 6441] 00000008:00010000:0.0:1789760775.186545:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.186551:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760775.186564:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26382336, 26386432) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.186579:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26382336=0x1929000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.186642:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.186645:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26386432 00000020:00010000:0.0:1789760775.186654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.186658:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.186661:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.186664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.186667:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.186687:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.186694:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.186713:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760775.186851:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26382336<>26386432 00000100:00100000:0.0:1789760775.186901:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760775.186919:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907456:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.220415:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973758 gen 2 00000008:00010000:0.0:1789760775.220481:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760775.220489:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.220500:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.220566:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760775.220600:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907456:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.221003:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6441, 6441] 00000080:00200000:1.0:1789760775.221020:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.221026:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.221044:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26386432 i_blocks 53257 00000020:00200000:1.0:1789760775.221048:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.221058:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.221095:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26386432, count: 4096 00000080:00200000:1.0:1789760775.221103:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26386432, bytes: 4096 00000080:00200000:1.0:1789760775.221114:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.221123:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.221125:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26386432, 26390528) 0 0 00000080:00200000:1.0:1789760775.221130:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.221147:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6442, 6442] 00020000:00200000:1.0:1789760775.221155:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26386432 chunk: [26386432, 26390528) 26390528, 4096 00020000:00200000:1.0:1789760775.221160:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.221208:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6442, 6442] 00000008:00010000:1.0:1789760775.221227:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.221232:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.221243:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26386432, 26390528) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.221257:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26386432=0x192a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.221316:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.221318:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26390528 00000020:00010000:1.0:1789760775.221327:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.221331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.221451:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.221454:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.221458:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.221467:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.221475:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.221493:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760775.221620:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 26386432<>26390528 00000100:00100000:0.0:1789760775.221685:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907584:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.281927:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973759 gen 2 00000008:00010000:0.0:1789760775.281973:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760775.281981:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.281993:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.282016:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760775.282045:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907584:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.282208:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6442, 6442] 00000080:00200000:1.0:1789760775.282227:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.282234:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.282249:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26390528 i_blocks 53257 00000020:00200000:1.0:1789760775.282253:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.282262:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.282901:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26390528, count: 4096 00000080:00200000:1.0:1789760775.282914:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26390528, bytes: 4096 00000080:00200000:1.0:1789760775.282926:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.282935:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.282937:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26390528, 26394624) 0 0 00000080:00200000:1.0:1789760775.282942:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.282973:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6443, 6443] 00020000:00200000:1.0:1789760775.282983:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26390528 chunk: [26390528, 26394624) 26394624, 4096 00020000:00200000:1.0:1789760775.282988:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.283011:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6443, 6443] 00000008:00010000:1.0:1789760775.283032:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.283037:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.283049:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26390528, 26394624) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.283064:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26390528=0x192b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.283131:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.283134:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26394624 00000020:00010000:1.0:1789760775.283142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.283145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.283148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.283151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.283154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.283162:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.283190:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.283208:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760775.283348:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 26390528<>26394624 00000100:00100000:0.0:1789760775.283874:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760775.283898:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907712:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.317153:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973760 gen 2 00000008:00010000:0.0:1789760775.317228:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760775.317236:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.317246:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.317303:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760775.317334:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907712:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.317492:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6443, 6443] 00000080:00200000:1.0:1789760775.317510:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.317516:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.317655:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26394624 i_blocks 53257 00000020:00200000:1.0:1789760775.317660:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.317669:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.317763:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26394624, count: 4096 00000080:00200000:1.0:1789760775.317771:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26394624, bytes: 4096 00000080:00200000:1.0:1789760775.317782:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.317791:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.317793:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26394624, 26398720) 0 0 00000080:00200000:1.0:1789760775.317797:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.317813:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6444, 6444] 00020000:00200000:1.0:1789760775.317822:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26394624 chunk: [26394624, 26398720) 26398720, 4096 00020000:00200000:1.0:1789760775.317826:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.317845:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6444, 6444] 00000008:00010000:1.0:1789760775.317863:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.317869:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.317881:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26394624, 26398720) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.317896:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26394624=0x192c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.317953:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.317956:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26398720 00000020:00010000:1.0:1789760775.317964:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.317968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.317971:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.317974:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.317978:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.317985:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.317992:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.318165:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760775.318428:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 26394624<>26398720 00000100:00100000:0.0:1789760775.318488:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907840:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.345005:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973761 gen 2 00000008:00010000:0.0:1789760775.345056:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760775.345065:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.345077:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.345103:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760775.345133:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495907840:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760775.345659:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6444, 6444] 00000080:00200000:3.0:1789760775.345679:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.345685:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.345702:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26398720 i_blocks 53257 00000020:00200000:3.0:1789760775.345707:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.345718:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.345753:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26398720, count: 4096 00000080:00200000:3.0:1789760775.345760:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26398720, bytes: 4096 00000080:00200000:3.0:1789760775.345772:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.345781:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760775.345783:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26398720, 26402816) 0 0 00000080:00200000:3.0:1789760775.345789:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.345806:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6445, 6445] 00020000:00200000:3.0:1789760775.345813:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26398720 chunk: [26398720, 26402816) 26402816, 4096 00020000:00200000:3.0:1789760775.345818:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.345838:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6445, 6445] 00000008:00010000:3.0:1789760775.345856:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.345862:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760775.345874:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26398720, 26402816) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.345889:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26398720=0x192d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.345950:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.345953:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26402816 00000020:00010000:3.0:1789760775.345961:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.345965:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.345969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.345972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.345974:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.345982:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.345988:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.346005:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760775.346133:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 26398720<>26402816 00000100:00100000:2.0:1789760775.346606:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760775.346628:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495907968:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760775.402081:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973762 gen 2 00000008:00010000:2.0:1789760775.402135:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760775.402143:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760775.402157:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760775.402206:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760775.402237:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495907968:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760775.402384:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6445, 6445] 00000080:00200000:3.0:1789760775.402401:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.402407:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.402422:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26402816 i_blocks 53257 00000020:00200000:3.0:1789760775.402426:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.402439:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.402473:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26402816, count: 4096 00000080:00200000:3.0:1789760775.402480:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26402816, bytes: 4096 00000080:00200000:3.0:1789760775.402490:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.402498:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760775.402501:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26402816, 26406912) 0 0 00000080:00200000:3.0:1789760775.402506:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.402526:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6446, 6446] 00020000:00200000:3.0:1789760775.402534:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26402816 chunk: [26402816, 26406912) 26406912, 4096 00020000:00200000:3.0:1789760775.402539:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.402558:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6446, 6446] 00000008:00010000:3.0:1789760775.402576:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.402581:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760775.402593:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26402816, 26406912) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.402607:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26402816=0x192e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.402664:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.402667:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26406912 00000020:00010000:3.0:1789760775.402674:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.402678:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.402681:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.402684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.402687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.402693:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.402699:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.402714:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760775.402829:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 26402816<>26406912 00000100:00100000:2.0:1789760775.403018:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495908096:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760775.454534:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973763 gen 2 00000008:00010000:2.0:1789760775.454586:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760775.454594:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760775.454606:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760775.454630:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760775.454662:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495908096:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760775.455133:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6446, 6446] 00000080:00200000:3.0:1789760775.455152:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.455159:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.455335:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26406912 i_blocks 53257 00000020:00200000:3.0:1789760775.455415:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.456918:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.456972:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26406912, count: 4096 00000080:00200000:2.0:1789760775.456984:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26406912, bytes: 4096 00000080:00200000:2.0:1789760775.456999:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.457009:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.457011:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26406912, 26411008) 0 0 00000080:00200000:2.0:1789760775.457017:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.457037:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6447, 6447] 00020000:00200000:2.0:1789760775.457047:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26406912 chunk: [26406912, 26411008) 26411008, 4096 00020000:00200000:2.0:1789760775.457051:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.457074:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6447, 6447] 00000008:00010000:2.0:1789760775.457093:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.457098:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760775.457479:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26406912, 26411008) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.457497:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26406912=0x192f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.457564:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.457567:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26411008 00000020:00010000:2.0:1789760775.457576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.457580:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.457583:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.457587:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.457590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.457598:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.457604:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.457670:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760775.457803:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26406912<>26411008 00000100:00100000:2.0:1789760775.457868:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760775.457895:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495908224:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760775.504812:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973764 gen 2 00000008:00010000:2.0:1789760775.504863:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760775.504872:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760775.504884:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760775.504956:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760775.504988:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495908224:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760775.505272:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6447, 6447] 00000080:00200000:3.0:1789760775.505289:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.505295:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.505508:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26411008 i_blocks 53257 00000020:00200000:2.0:1789760775.505513:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.505593:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.505631:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26411008, count: 4096 00000080:00200000:2.0:1789760775.505639:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26411008, bytes: 4096 00000080:00200000:2.0:1789760775.505650:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.505659:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.505662:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26411008, 26415104) 0 0 00000080:00200000:2.0:1789760775.505666:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.505758:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6448, 6448] 00020000:00200000:2.0:1789760775.505768:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26411008 chunk: [26411008, 26415104) 26415104, 4096 00020000:00200000:2.0:1789760775.505773:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.505795:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6448, 6448] 00000008:00010000:2.0:1789760775.505813:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.505819:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760775.505832:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26411008, 26415104) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.505846:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26411008=0x1930000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.505904:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.505907:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26415104 00000020:00010000:2.0:1789760775.505916:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.505919:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.505922:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.505926:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.505929:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.505936:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.505943:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.505993:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760775.506116:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26411008<>26415104 00000100:00100000:3.0:1789760775.506200:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495908352:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760775.552452:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973765 gen 2 00000008:00010000:2.0:1789760775.553246:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760775.553255:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760775.553268:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760775.553604:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760775.553635:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495908352:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760775.553654:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6448, 6448] 00000080:00200000:0.0:1789760775.553674:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.553681:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.553698:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26415104 i_blocks 53257 00000020:00200000:0.0:1789760775.553702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.553713:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.553748:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26415104, count: 4096 00000080:00200000:0.0:1789760775.553755:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26415104, bytes: 4096 00000080:00200000:0.0:1789760775.553768:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.553776:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.553778:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26415104, 26419200) 0 0 00000080:00200000:0.0:1789760775.553783:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.553798:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6449, 6449] 00020000:00200000:0.0:1789760775.553807:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26415104 chunk: [26415104, 26419200) 26419200, 4096 00020000:00200000:0.0:1789760775.553811:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.553831:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6449, 6449] 00000008:00010000:0.0:1789760775.553848:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.553854:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760775.553864:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26415104, 26419200) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.553878:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26415104=0x1931000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.553946:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.553949:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26419200 00000020:00010000:0.0:1789760775.553958:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.553961:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.553964:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.553967:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.553970:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.553977:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.553984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.554001:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760775.554841:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26415104<>26419200 00000100:00100000:1.0:1789760775.554919:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495908480:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.613471:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973766 gen 2 00000008:00010000:0.0:1789760775.613529:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760775.613539:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.613550:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.613571:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760775.613601:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495908480:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.613762:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6449, 6449] 00000080:00200000:1.0:1789760775.613780:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.613787:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.613803:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26419200 i_blocks 53257 00000020:00200000:1.0:1789760775.613807:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.613817:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.613854:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26419200, count: 4096 00000080:00200000:1.0:1789760775.613861:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26419200, bytes: 4096 00000080:00200000:1.0:1789760775.613873:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.613882:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.613884:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26419200, 26423296) 0 0 00000080:00200000:1.0:1789760775.613888:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.613905:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6450, 6450] 00020000:00200000:1.0:1789760775.613912:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26419200 chunk: [26419200, 26423296) 26423296, 4096 00020000:00200000:1.0:1789760775.613917:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.613937:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6450, 6450] 00000008:00010000:1.0:1789760775.613956:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.613962:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.613974:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26419200, 26423296) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.613988:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26419200=0x1932000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.614050:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.614053:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26423296 00000020:00010000:1.0:1789760775.614061:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.614065:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.614068:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.614072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.614076:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.614084:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.614092:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.614108:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760775.614275:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 26419200<>26423296 00000100:00100000:0.0:1789760775.614797:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495908608:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760775.628231:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:0.0:1789760775.663758:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760775.663776:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.663789:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.663804:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760775.663836:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495908608:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.663959:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6450, 6450] 00000080:00200000:1.0:1789760775.663977:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.663984:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.664001:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26423296 i_blocks 53257 00000020:00200000:1.0:1789760775.664005:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.664014:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.664053:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26423296, count: 4096 00000080:00200000:1.0:1789760775.664061:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26423296, bytes: 4096 00000080:00200000:1.0:1789760775.664073:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.664083:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.664085:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26423296, 26427392) 0 0 00000080:00200000:1.0:1789760775.664090:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.664109:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6451, 6451] 00020000:00200000:1.0:1789760775.664116:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26423296 chunk: [26423296, 26427392) 26427392, 4096 00020000:00200000:1.0:1789760775.664121:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.664142:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6451, 6451] 00000008:00010000:1.0:1789760775.664162:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.664617:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.664636:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26423296, 26427392) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.664651:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26423296=0x1933000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.664788:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.664792:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26427392 00000020:00010000:1.0:1789760775.664800:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.664804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.664807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.664811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.664815:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.664827:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.664835:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.664858:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760775.664999:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 26423296<>26427392 00000100:00100000:0.0:1789760775.665829:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760775.665871:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495908736:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.707814:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973768 gen 2 00000008:00010000:0.0:1789760775.708720:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760775.708732:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.708744:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.708767:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760775.708800:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495908736:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760775.709045:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6451, 6451] 00000080:00200000:2.0:1789760775.709062:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.709068:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.709085:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26427392 i_blocks 53257 00000020:00200000:2.0:1789760775.709088:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.709098:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.709136:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26427392, count: 4096 00000080:00200000:2.0:1789760775.709143:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26427392, bytes: 4096 00000080:00200000:2.0:1789760775.709155:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.709164:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.709166:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26427392, 26431488) 0 0 00000080:00200000:2.0:1789760775.709185:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.709202:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6452, 6452] 00020000:00200000:2.0:1789760775.709210:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26427392 chunk: [26427392, 26431488) 26431488, 4096 00020000:00200000:2.0:1789760775.709214:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.709235:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6452, 6452] 00000008:00010000:2.0:1789760775.709255:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.709261:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760775.709274:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26427392, 26431488) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.709289:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26427392=0x1934000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.710629:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.710640:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26431488 00000020:00010000:2.0:1789760775.710654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.710658:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.710664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.710667:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.710670:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.710753:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.710763:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.710792:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760775.711101:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26427392<>26431488 00000100:00100000:3.0:1789760775.711280:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495908864:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760775.764581:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973769 gen 2 00000008:00010000:3.0:1789760775.764636:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760775.764645:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.764656:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.764679:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760775.764717:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495908864:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760775.765079:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6452, 6452] 00000080:00200000:2.0:1789760775.765101:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.765107:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.765123:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26431488 i_blocks 53257 00000020:00200000:2.0:1789760775.765127:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.765140:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.765199:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26431488, count: 4096 00000080:00200000:2.0:1789760775.765206:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26431488, bytes: 4096 00000080:00200000:2.0:1789760775.765217:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.765225:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.765227:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26431488, 26435584) 0 0 00000080:00200000:2.0:1789760775.765231:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.765243:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6453, 6453] 00020000:00200000:2.0:1789760775.765251:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26431488 chunk: [26431488, 26435584) 26435584, 4096 00020000:00200000:2.0:1789760775.765254:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.765272:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6453, 6453] 00000008:00010000:2.0:1789760775.765285:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.765290:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760775.765300:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26431488, 26435584) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.765311:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26431488=0x1935000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.765372:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.765375:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26435584 00000020:00010000:2.0:1789760775.765381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.765384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.765388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.765391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.765394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.765402:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.765407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.765425:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760775.765528:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 26431488<>26435584 00000100:00100000:3.0:1789760775.766461:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760775.766486:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495908992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760775.805756:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973770 gen 2 00000008:00010000:3.0:1789760775.805794:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760775.805800:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.805810:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.805834:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760775.805861:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495908992:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760775.805964:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6453, 6453] 00000080:00200000:2.0:1789760775.805977:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.805982:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.805996:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26435584 i_blocks 53257 00000020:00200000:2.0:1789760775.806000:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.806008:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.806032:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26435584, count: 4096 00000080:00200000:2.0:1789760775.806038:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26435584, bytes: 4096 00000080:00200000:2.0:1789760775.806046:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.806053:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.806055:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26435584, 26439680) 0 0 00000080:00200000:2.0:1789760775.806059:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.806072:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6454, 6454] 00020000:00200000:2.0:1789760775.806079:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26435584 chunk: [26435584, 26439680) 26439680, 4096 00020000:00200000:2.0:1789760775.806083:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.806101:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6454, 6454] 00000008:00010000:2.0:1789760775.806127:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.806133:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760775.806142:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26435584, 26439680) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.806154:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26435584=0x1936000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.806233:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.806236:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26439680 00000020:00010000:2.0:1789760775.806253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.806256:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.806266:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.806270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.806272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.806279:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.806285:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.806298:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760775.806400:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 26435584<>26439680 00000100:00100000:3.0:1789760775.806486:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760775.806500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495909120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760775.834922:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973771 gen 2 00000008:00010000:3.0:1789760775.834973:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760775.834981:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.834992:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.835018:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760775.835048:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495909120:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760775.835269:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6454, 6454] 00000080:00200000:2.0:1789760775.835284:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.835289:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.835302:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26439680 i_blocks 53257 00000020:00200000:2.0:1789760775.835304:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.835312:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.835344:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26439680, count: 4096 00000080:00200000:2.0:1789760775.835360:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26439680, bytes: 4096 00000080:00200000:2.0:1789760775.835371:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.835380:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.835382:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26439680, 26443776) 0 0 00000080:00200000:2.0:1789760775.835387:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.835403:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6455, 6455] 00020000:00200000:2.0:1789760775.835412:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26439680 chunk: [26439680, 26443776) 26443776, 4096 00020000:00200000:2.0:1789760775.835416:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.835435:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6455, 6455] 00000008:00010000:2.0:1789760775.835453:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.835458:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760775.835470:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26439680, 26443776) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.835496:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26439680=0x1937000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.835551:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.835553:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26443776 00000020:00010000:2.0:1789760775.835561:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.835563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.835566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.835569:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.835571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.835576:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.835581:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.835594:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760775.836462:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 26439680<>26443776 00000100:00100000:3.0:1789760775.836829:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495909248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760775.873635:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973772 gen 2 00000008:00010000:3.0:1789760775.873673:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760775.873680:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.873690:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.873711:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760775.873737:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495909248:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.873842:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6455, 6455] 00000080:00200000:1.0:1789760775.873855:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.873861:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.873873:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26443776 i_blocks 53257 00000020:00200000:1.0:1789760775.873877:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.873885:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.873910:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26443776, count: 4096 00000080:00200000:1.0:1789760775.873916:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26443776, bytes: 4096 00000080:00200000:1.0:1789760775.873924:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.873933:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.873934:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26443776, 26447872) 0 0 00000080:00200000:1.0:1789760775.873938:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.874016:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6456, 6456] 00020000:00200000:1.0:1789760775.874025:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26443776 chunk: [26443776, 26447872) 26447872, 4096 00020000:00200000:1.0:1789760775.874029:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.874044:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6456, 6456] 00000008:00010000:1.0:1789760775.874058:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.874063:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.874072:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26443776, 26447872) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.874082:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26443776=0x1938000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.874207:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.874211:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26447872 00000020:00010000:1.0:1789760775.874218:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.874221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.874224:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.874227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.874230:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.874237:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.874242:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.874256:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760775.874343:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 26443776<>26447872 00000100:00100000:0.0:1789760775.875212:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760775.875232:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495909376:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.918366:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973773 gen 2 00000008:00010000:0.0:1789760775.918403:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760775.918410:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.918419:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.918436:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760775.918460:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495909376:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.918551:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6456, 6456] 00000080:00200000:1.0:1789760775.918563:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.918568:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.918580:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26447872 i_blocks 53257 00000020:00200000:1.0:1789760775.918583:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.918592:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.918616:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26447872, count: 4096 00000080:00200000:1.0:1789760775.918623:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26447872, bytes: 4096 00000080:00200000:1.0:1789760775.918630:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.918638:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.918640:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26447872, 26451968) 0 0 00000080:00200000:1.0:1789760775.918644:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.918657:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6457, 6457] 00020000:00200000:1.0:1789760775.918665:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26447872 chunk: [26447872, 26451968) 26451968, 4096 00020000:00200000:1.0:1789760775.918669:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.918684:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6457, 6457] 00000008:00010000:1.0:1789760775.918697:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.918702:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.918711:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26447872, 26451968) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.918720:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26447872=0x1939000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.918760:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.918762:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26451968 00000020:00010000:1.0:1789760775.918768:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.918772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.918774:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.918778:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.918781:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.918786:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.918792:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.918804:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760775.918894:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 26447872<>26451968 00000100:00100000:0.0:1789760775.918979:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760775.918993:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495909504:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.949835:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973774 gen 2 00000008:00010000:0.0:1789760775.949873:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760775.949879:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.949887:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.949946:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760775.949969:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495909504:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.950167:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6457, 6457] 00000080:00200000:1.0:1789760775.950205:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.950211:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.950224:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26451968 i_blocks 53257 00000020:00200000:1.0:1789760775.950227:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.950235:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.950261:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26451968, count: 4096 00000080:00200000:1.0:1789760775.950267:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26451968, bytes: 4096 00000080:00200000:1.0:1789760775.950274:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.950281:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.950283:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26451968, 26456064) 0 0 00000080:00200000:1.0:1789760775.950287:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.950300:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6458, 6458] 00020000:00200000:1.0:1789760775.950307:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26451968 chunk: [26451968, 26456064) 26456064, 4096 00020000:00200000:1.0:1789760775.950311:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.950326:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6458, 6458] 00000008:00010000:1.0:1789760775.950338:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.950344:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.950363:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26451968, 26456064) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.950375:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26451968=0x193a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.951284:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.951290:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26456064 00000020:00010000:1.0:1789760775.951300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.951304:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.951307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.951310:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.951315:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.951322:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.951328:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.951348:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760775.951517:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 26451968<>26456064 00000100:00100000:0.0:1789760775.951603:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495909632:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760775.991051:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973775 gen 2 00000008:00010000:0.0:1789760775.991115:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760775.991124:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.991135:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.991156:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760775.991225:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495909632:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760775.991473:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6458, 6458] 00000080:00200000:1.0:1789760775.991491:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.991497:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.991513:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26456064 i_blocks 53257 00000020:00200000:1.0:1789760775.991588:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.991599:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.991634:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26456064, count: 4096 00000080:00200000:1.0:1789760775.991640:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26456064, bytes: 4096 00000080:00200000:1.0:1789760775.991650:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.991659:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.991661:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26456064, 26460160) 0 0 00000080:00200000:1.0:1789760775.991665:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.991748:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6459, 6459] 00020000:00200000:1.0:1789760775.991756:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26456064 chunk: [26456064, 26460160) 26460160, 4096 00020000:00200000:1.0:1789760775.991760:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.991781:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6459, 6459] 00000008:00010000:1.0:1789760775.991798:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.991805:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760775.991818:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26456064, 26460160) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.991831:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26456064=0x193b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.992423:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.992428:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26460160 00000020:00010000:1.0:1789760775.992438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.992443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.992446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.992450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.992453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.992461:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.992469:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.992493:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760775.992641:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 26456064<>26460160 00000100:00100000:0.0:1789760775.992865:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495909760:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760776.038688:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760776.038704:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.038716:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.038780:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760776.038811:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495909760:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.038960:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6459, 6459] 00000080:00200000:1.0:1789760776.038977:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.038984:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.039263:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26460160 i_blocks 53257 00000020:00200000:1.0:1789760776.039268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.039278:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.039313:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26460160, count: 4096 00000080:00200000:1.0:1789760776.039319:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26460160, bytes: 4096 00000080:00200000:1.0:1789760776.039330:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.039339:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.039341:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26460160, 26464256) 0 0 00000080:00200000:1.0:1789760776.039346:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.039375:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6460, 6460] 00020000:00200000:1.0:1789760776.039383:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26460160 chunk: [26460160, 26464256) 26464256, 4096 00020000:00200000:1.0:1789760776.039388:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.039409:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6460, 6460] 00000008:00010000:1.0:1789760776.039430:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.039436:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.039448:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26460160, 26464256) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.039850:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26460160=0x193c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.039971:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.039975:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26464256 00000020:00010000:1.0:1789760776.039983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.039988:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.039991:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.039994:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.039998:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.040006:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.040013:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.040030:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760776.040157:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 26460160<>26464256 00000100:00100000:1.0:1789760776.040723:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495909888:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.081820:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973777 gen 2 00000008:00010000:0.0:1789760776.081879:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760776.081887:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.081897:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.081920:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000080:00200000:1.0:1789760776.082100:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6460, 6460] 00000080:00200000:1.0:1789760776.082120:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.082126:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.082141:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26464256 i_blocks 53257 00000020:00200000:1.0:1789760776.082145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.082154:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:0.0:1789760776.082474:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495909888:192.168.203.109@tcp:4:dd.0 00000080:00200008:1.0:1789760776.082683:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26464256, count: 4096 00000080:00200000:1.0:1789760776.082692:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26464256, bytes: 4096 00000080:00200000:1.0:1789760776.082705:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.082725:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.082728:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26464256, 26468352) 0 0 00000080:00200000:1.0:1789760776.082734:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.082750:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6461, 6461] 00020000:00200000:1.0:1789760776.082759:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26464256 chunk: [26464256, 26468352) 26468352, 4096 00020000:00200000:1.0:1789760776.082764:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.082784:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6461, 6461] 00000008:00010000:1.0:1789760776.082803:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.082809:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.082821:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26464256, 26468352) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.082836:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26464256=0x193d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.082893:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.082895:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26468352 00000020:00010000:1.0:1789760776.082904:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.082908:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.082912:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.082915:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.082918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.082925:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.082932:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.082950:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760776.083076:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 26464256<>26468352 00000100:00100000:0.0:1789760776.083264:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495910016:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.119482:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973778 gen 2 00000008:00010000:0.0:1789760776.119603:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760776.119612:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.119623:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.119686:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760776.119716:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495910016:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.120220:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6461, 6461] 00000080:00200000:1.0:1789760776.120239:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.120247:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.120263:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26468352 i_blocks 53257 00000020:00200000:1.0:1789760776.120267:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.120278:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.120312:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26468352, count: 4096 00000080:00200000:1.0:1789760776.120320:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26468352, bytes: 4096 00000080:00200000:1.0:1789760776.120332:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.120341:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.120344:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26468352, 26472448) 0 0 00000080:00200000:1.0:1789760776.120918:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.120937:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6462, 6462] 00020000:00200000:1.0:1789760776.120946:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26468352 chunk: [26468352, 26472448) 26472448, 4096 00020000:00200000:1.0:1789760776.120950:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.120971:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6462, 6462] 00000008:00010000:1.0:1789760776.120990:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.120996:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.122148:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26468352, 26472448) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.122185:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26468352=0x193e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.122288:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.122291:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26472448 00000020:00010000:1.0:1789760776.122302:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.122305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.122309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.122312:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.122315:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.122326:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.122335:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.122494:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760776.122690:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 26468352<>26472448 00000100:00100000:0.0:1789760776.122883:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760776.122911:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495910144:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.160244:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973779 gen 2 00000008:00010000:0.0:1789760776.160299:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760776.160308:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.160320:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.160340:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760776.160381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495910144:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.160548:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6462, 6462] 00000080:00200000:1.0:1789760776.160565:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.160572:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.160588:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26472448 i_blocks 53257 00000020:00200000:1.0:1789760776.160592:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.160601:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.160636:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26472448, count: 4096 00000080:00200000:1.0:1789760776.160643:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26472448, bytes: 4096 00000080:00200000:1.0:1789760776.160653:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.160663:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.160665:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26472448, 26476544) 0 0 00000080:00200000:1.0:1789760776.160736:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.160754:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6463, 6463] 00020000:00200000:1.0:1789760776.160763:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26472448 chunk: [26472448, 26476544) 26476544, 4096 00020000:00200000:1.0:1789760776.160767:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.160788:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6463, 6463] 00000008:00010000:1.0:1789760776.160806:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.160812:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.160824:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26472448, 26476544) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.160838:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26472448=0x193f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.160908:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.160911:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26476544 00000020:00010000:1.0:1789760776.160920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.160924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.160928:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.160931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd205fddb78 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.160934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.160942:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.160948:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.160965:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760776.163012:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 26472448<>26476544 00000100:00100000:0.0:1789760776.163093:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495910272:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760776.199437:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760776.199455:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.199466:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.199582:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760776.199616:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495910272:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760776.199690:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6463, 6463] 00000080:00200000:3.0:1789760776.199744:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.199751:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.199767:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26476544 i_blocks 53257 00000020:00200000:3.0:1789760776.199771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.199780:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.199830:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26476544, count: 4096 00000080:00200000:3.0:1789760776.199839:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26476544, bytes: 4096 00000080:00200000:3.0:1789760776.199850:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.199859:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760776.199862:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26476544, 26480640) 0 0 00000080:00200000:3.0:1789760776.199867:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.199884:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6464, 6464] 00020000:00200000:3.0:1789760776.199892:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26476544 chunk: [26476544, 26480640) 26480640, 4096 00020000:00200000:3.0:1789760776.199897:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.199918:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6464, 6464] 00000008:00010000:3.0:1789760776.199936:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.199941:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760776.199952:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26476544, 26480640) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.199967:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26476544=0x1940000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.200113:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.200116:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26480640 00000020:00010000:3.0:1789760776.200126:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.200130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.200134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.200138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.200141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.200149:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.200156:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.200333:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760776.200471:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 26476544<>26480640 00000100:00100000:3.0:1789760776.200609:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495910400:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760776.231417:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973781 gen 2 00000008:00010000:3.0:1789760776.231485:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:3.0:1789760776.231494:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.231506:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.231529:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000080:00200000:2.0:1789760776.231776:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6464, 6464] 00000080:00200000:2.0:1789760776.231796:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.231802:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.231819:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26480640 i_blocks 53257 00000020:00200000:2.0:1789760776.231823:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.231833:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.231868:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26480640, count: 4096 00000080:00200000:2.0:1789760776.231877:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26480640, bytes: 4096 00000080:00200000:2.0:1789760776.231888:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.231897:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.231899:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26480640, 26484736) 0 0 00000080:00200000:2.0:1789760776.231903:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.231921:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6465, 6465] 00020000:00200000:2.0:1789760776.231929:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26480640 chunk: [26480640, 26484736) 26484736, 4096 00020000:00200000:2.0:1789760776.231934:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.231954:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6465, 6465] 00000008:00010000:2.0:1789760776.231973:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.231979:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760776.231991:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26480640, 26484736) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.232005:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26480640=0x1941000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.232057:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.232060:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26484736 00000020:00010000:2.0:1789760776.232069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.232072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.232076:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.232079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.232082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.232089:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.232096:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760776.236056:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495910400:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760776.236141:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:3.0:1789760776.236314:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 26480640<>26484736 00000100:00100000:2.0:1789760776.236401:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760776.236488:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495910528:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760776.298612:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973782 gen 2 00000008:00010000:2.0:1789760776.298662:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760776.298670:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.298682:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.298708:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760776.298744:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495910528:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760776.298860:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6465, 6465] 00000080:00200000:0.0:1789760776.298878:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760776.298884:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760776.298900:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26484736 i_blocks 53257 00000020:00200000:0.0:1789760776.298904:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760776.298914:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760776.298948:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26484736, count: 4096 00000080:00200000:0.0:1789760776.298955:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26484736, bytes: 4096 00000080:00200000:0.0:1789760776.298966:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760776.298975:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760776.298977:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26484736, 26488832) 0 0 00000080:00200000:0.0:1789760776.298982:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760776.298998:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6466, 6466] 00020000:00200000:0.0:1789760776.299006:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26484736 chunk: [26484736, 26488832) 26488832, 4096 00020000:00200000:0.0:1789760776.299011:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760776.299031:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6466, 6466] 00000008:00010000:0.0:1789760776.299049:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760776.299055:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760776.299066:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26484736, 26488832) append 0 append lockpos 0 00000080:00200000:0.0:1789760776.299081:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26484736=0x1942000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760776.299139:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760776.299142:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26488832 00000020:00010000:0.0:1789760776.299150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760776.299154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.299157:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.299161:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760776.299164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760776.299217:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760776.299225:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760776.299240:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760776.299377:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26484736<>26488832 00000100:00100000:0.0:1789760776.299414:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760776.299430:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495910656:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.351364:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973783 gen 2 00000008:00010000:0.0:1789760776.351417:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760776.351426:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.351436:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.351462:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760776.351492:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495910656:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760776.351605:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6466, 6466] 00000080:00200000:2.0:1789760776.351623:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.351630:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.351646:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26488832 i_blocks 53257 00000020:00200000:2.0:1789760776.351651:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.351660:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.351708:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26488832, count: 4096 00000080:00200000:2.0:1789760776.351715:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26488832, bytes: 4096 00000080:00200000:2.0:1789760776.351725:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.351734:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.351737:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26488832, 26492928) 0 0 00000080:00200000:2.0:1789760776.351741:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.351759:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6467, 6467] 00020000:00200000:2.0:1789760776.351768:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26488832 chunk: [26488832, 26492928) 26492928, 4096 00020000:00200000:2.0:1789760776.351772:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.351790:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6467, 6467] 00000008:00010000:2.0:1789760776.351808:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.351815:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760776.351826:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26488832, 26492928) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.351840:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26488832=0x1943000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.351896:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.351899:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26492928 00000020:00010000:2.0:1789760776.351907:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.351911:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.351915:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.351918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.351922:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.351929:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.351935:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.351952:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760776.352072:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 26488832<>26492928 00000100:00100000:3.0:1789760776.352201:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760776.352219:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495910784:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760776.401416:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760776.401432:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.401443:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.401466:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760776.401497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495910784:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760776.401817:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6467, 6467] 00000080:00200000:2.0:1789760776.401832:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.401838:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.401852:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26492928 i_blocks 53257 00000020:00200000:2.0:1789760776.401856:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.401865:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.401904:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26492928, count: 4096 00000080:00200000:2.0:1789760776.401913:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26492928, bytes: 4096 00000080:00200000:2.0:1789760776.401925:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.401934:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.401937:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26492928, 26497024) 0 0 00000080:00200000:2.0:1789760776.401941:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.401957:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6468, 6468] 00020000:00200000:2.0:1789760776.401965:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26492928 chunk: [26492928, 26497024) 26497024, 4096 00020000:00200000:2.0:1789760776.401969:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.401989:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6468, 6468] 00000008:00010000:2.0:1789760776.402007:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.402014:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760776.402027:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26492928, 26497024) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.402042:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26492928=0x1944000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.402096:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.402099:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26497024 00000020:00010000:2.0:1789760776.402107:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.402111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.402114:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.402119:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.402122:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.402130:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.402137:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.402153:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760776.402294:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 26492928<>26497024 00000100:00100000:3.0:1789760776.402633:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760776.402653:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495910912:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760776.455317:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973785 gen 2 00000008:00010000:3.0:1789760776.455398:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760776.455407:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.455417:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.455440:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760776.455471:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495910912:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760776.455665:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6468, 6468] 00000080:00200000:2.0:1789760776.455692:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.455698:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.455715:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26497024 i_blocks 53257 00000020:00200000:2.0:1789760776.455719:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.455729:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.455764:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26497024, count: 4096 00000080:00200000:2.0:1789760776.455772:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26497024, bytes: 4096 00000080:00200000:2.0:1789760776.455782:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.455791:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.455794:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26497024, 26501120) 0 0 00000080:00200000:2.0:1789760776.455798:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.455814:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6469, 6469] 00020000:00200000:2.0:1789760776.455822:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26497024 chunk: [26497024, 26501120) 26501120, 4096 00020000:00200000:2.0:1789760776.455827:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.455846:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6469, 6469] 00000008:00010000:2.0:1789760776.455865:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.455871:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760776.455882:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26497024, 26501120) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.455896:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26497024=0x1945000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.455952:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.455955:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26501120 00000020:00010000:2.0:1789760776.455963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.455967:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.455970:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.455973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.455976:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.455982:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.455989:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.456006:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760776.456134:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 26497024<>26501120 00000100:00100000:3.0:1789760776.456201:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495911040:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760776.506125:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973786 gen 2 00000008:00010000:3.0:1789760776.506300:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760776.506309:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.506321:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.506483:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760776.506512:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495911040:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.506662:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6469, 6469] 00000080:00200000:1.0:1789760776.506740:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.506748:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.506765:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26501120 i_blocks 53257 00000020:00200000:1.0:1789760776.506769:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.506779:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.506816:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26501120, count: 4096 00000080:00200000:1.0:1789760776.506824:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26501120, bytes: 4096 00000080:00200000:1.0:1789760776.506835:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.506844:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.506846:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26501120, 26505216) 0 0 00000080:00200000:1.0:1789760776.506910:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.506929:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6470, 6470] 00020000:00200000:1.0:1789760776.506938:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26501120 chunk: [26501120, 26505216) 26505216, 4096 00020000:00200000:1.0:1789760776.506943:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.506965:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6470, 6470] 00000008:00010000:1.0:1789760776.506984:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.506990:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.507002:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26501120, 26505216) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.507015:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26501120=0x1946000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.507073:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.507076:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26505216 00000020:00010000:1.0:1789760776.507085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.507089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.507093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.507097:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.507101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.507109:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.507116:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.507134:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760776.507277:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26501120<>26505216 00000100:00100000:0.0:1789760776.507373:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495911168:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.541500:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973787 gen 2 00000008:00010000:0.0:1789760776.541561:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760776.541571:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.541581:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.541599:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760776.541628:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495911168:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.542209:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6470, 6470] 00000080:00200000:1.0:1789760776.542226:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.542233:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.542249:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26505216 i_blocks 53257 00000020:00200000:1.0:1789760776.542253:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.542262:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.542296:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26505216, count: 4096 00000080:00200000:1.0:1789760776.542303:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26505216, bytes: 4096 00000080:00200000:1.0:1789760776.542313:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.542322:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.542325:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26505216, 26509312) 0 0 00000080:00200000:1.0:1789760776.542329:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.542355:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6471, 6471] 00020000:00200000:1.0:1789760776.542365:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26505216 chunk: [26505216, 26509312) 26509312, 4096 00020000:00200000:1.0:1789760776.542370:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.542390:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6471, 6471] 00000008:00010000:1.0:1789760776.542407:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.542412:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.542424:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26505216, 26509312) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.542438:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26505216=0x1947000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.542494:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.542497:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26509312 00000020:00010000:1.0:1789760776.542505:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.542509:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.542513:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.542517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.542520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.542528:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.542545:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.542562:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760776.542696:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 26505216<>26509312 00000100:00100000:0.0:1789760776.542952:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495911296:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.569412:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973788 gen 2 00000008:00010000:0.0:1789760776.569465:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760776.569474:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.569486:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.569573:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760776.569601:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495911296:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.569750:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6471, 6471] 00000080:00200000:1.0:1789760776.569768:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.569773:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.569790:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26509312 i_blocks 53257 00000020:00200000:1.0:1789760776.569794:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.569803:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.569839:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26509312, count: 4096 00000080:00200000:1.0:1789760776.569846:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26509312, bytes: 4096 00000080:00200000:1.0:1789760776.569933:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.569942:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.569945:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26509312, 26513408) 0 0 00000080:00200000:1.0:1789760776.569949:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.569963:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6472, 6472] 00020000:00200000:1.0:1789760776.569972:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26509312 chunk: [26509312, 26513408) 26513408, 4096 00020000:00200000:1.0:1789760776.569976:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.569997:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6472, 6472] 00000008:00010000:1.0:1789760776.570015:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.570021:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.570042:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26509312, 26513408) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.570058:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26509312=0x1948000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.570280:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.570283:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26513408 00000020:00010000:1.0:1789760776.570292:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.570295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.570299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.570303:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.570306:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.570314:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.570479:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.570506:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760776.570634:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 26509312<>26513408 00000100:00100000:0.0:1789760776.571551:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495911424:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.609801:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973789 gen 2 00000008:00010000:0.0:1789760776.609914:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760776.609923:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.609935:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.610006:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760776.610044:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495911424:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.610167:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6472, 6472] 00000080:00200000:1.0:1789760776.610213:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.610229:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.610247:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26513408 i_blocks 53257 00000020:00200000:1.0:1789760776.610250:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.610260:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.610296:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26513408, count: 4096 00000080:00200000:1.0:1789760776.610303:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26513408, bytes: 4096 00000080:00200000:1.0:1789760776.610312:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.610321:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.610334:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26513408, 26517504) 0 0 00000080:00200000:1.0:1789760776.610338:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.610364:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6473, 6473] 00020000:00200000:1.0:1789760776.610373:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26513408 chunk: [26513408, 26517504) 26517504, 4096 00020000:00200000:1.0:1789760776.610378:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.610398:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6473, 6473] 00000008:00010000:1.0:1789760776.610415:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.610421:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.610431:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26513408, 26517504) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.610589:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26513408=0x1949000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.610669:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.610672:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26517504 00000020:00010000:1.0:1789760776.610682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.610686:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.610690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.610694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.610697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.610705:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.610712:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.610730:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760776.610851:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 26513408<>26517504 00000100:00100000:0.0:1789760776.610926:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495911552:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.646004:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973790 gen 2 00000008:00010000:0.0:1789760776.646054:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760776.646063:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.646073:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.646096:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760776.646127:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495911552:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.646387:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6473, 6473] 00000080:00200000:1.0:1789760776.646404:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.646411:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.646495:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26517504 i_blocks 53257 00000020:00200000:1.0:1789760776.646500:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.646510:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.646550:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26517504, count: 4096 00000080:00200000:1.0:1789760776.646557:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26517504, bytes: 4096 00000080:00200000:1.0:1789760776.646568:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.646577:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.646580:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26517504, 26521600) 0 0 00000080:00200000:1.0:1789760776.646583:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.646598:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6474, 6474] 00020000:00200000:1.0:1789760776.646605:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26517504 chunk: [26517504, 26521600) 26521600, 4096 00020000:00200000:1.0:1789760776.646608:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.646626:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6474, 6474] 00000008:00010000:1.0:1789760776.646639:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.646643:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.646654:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26517504, 26521600) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.646665:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26517504=0x194a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.646720:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.646723:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26521600 00000020:00010000:1.0:1789760776.646731:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.646735:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.646739:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.646743:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.646746:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.646753:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.646760:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.646776:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760776.646896:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 26517504<>26521600 00000100:00100000:0.0:1789760776.647074:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495911680:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760776.651748:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760776.671863:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973791 gen 2 00000008:00010000:0.0:1789760776.671921:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760776.671931:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.671944:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.671968:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760776.672005:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495911680:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.672310:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6474, 6474] 00000080:00200000:1.0:1789760776.672331:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.672338:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.672368:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26521600 i_blocks 53257 00000020:00200000:1.0:1789760776.672372:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.672383:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.672419:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26521600, count: 4096 00000080:00200000:1.0:1789760776.672426:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26521600, bytes: 4096 00000080:00200000:1.0:1789760776.672437:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.672446:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.672448:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26521600, 26525696) 0 0 00000080:00200000:1.0:1789760776.672452:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.672468:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6475, 6475] 00020000:00200000:1.0:1789760776.672477:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26521600 chunk: [26521600, 26525696) 26525696, 4096 00020000:00200000:1.0:1789760776.672482:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.672501:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6475, 6475] 00000008:00010000:1.0:1789760776.672518:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.672524:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.672535:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26521600, 26525696) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.672549:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26521600=0x194b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.672608:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.672611:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26525696 00000020:00010000:1.0:1789760776.672620:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.672623:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.672627:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.672631:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.672634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.672641:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.672648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.672675:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760776.672803:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 26521600<>26525696 00000100:00100000:0.0:1789760776.673061:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495911808:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.712946:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973792 gen 2 00000008:00010000:0.0:1789760776.713002:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760776.713011:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.713022:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.713043:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760776.713075:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495911808:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.713442:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6475, 6475] 00000080:00200000:1.0:1789760776.713460:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.713467:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.713483:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26525696 i_blocks 53257 00000020:00200000:1.0:1789760776.713487:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.713496:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.713535:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26525696, count: 4096 00000080:00200000:1.0:1789760776.713542:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26525696, bytes: 4096 00000080:00200000:1.0:1789760776.713552:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.713560:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.713563:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26525696, 26529792) 0 0 00000080:00200000:1.0:1789760776.713567:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.713586:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6476, 6476] 00020000:00200000:1.0:1789760776.713594:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26525696 chunk: [26525696, 26529792) 26529792, 4096 00020000:00200000:1.0:1789760776.713599:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.713621:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6476, 6476] 00000008:00010000:1.0:1789760776.713643:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.713649:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.713661:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26525696, 26529792) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.713736:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26525696=0x194c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.713797:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.713800:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26529792 00000020:00010000:1.0:1789760776.713809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.713812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.713816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.713820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.713823:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.713831:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.713837:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.713907:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760776.714039:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 26525696<>26529792 00000100:00100000:0.0:1789760776.714166:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495911936:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.745808:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973793 gen 2 00000008:00010000:0.0:1789760776.745853:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760776.745861:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.745873:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.745894:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760776.745925:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495911936:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.746035:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6476, 6476] 00000080:00200000:1.0:1789760776.746049:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.746053:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.746069:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26529792 i_blocks 53257 00000020:00200000:1.0:1789760776.746072:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.746082:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.746114:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26529792, count: 4096 00000080:00200000:1.0:1789760776.746121:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26529792, bytes: 4096 00000080:00200000:1.0:1789760776.746131:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.746139:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.746141:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26529792, 26533888) 0 0 00000080:00200000:1.0:1789760776.746145:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.746160:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6477, 6477] 00020000:00200000:1.0:1789760776.746197:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26529792 chunk: [26529792, 26533888) 26533888, 4096 00020000:00200000:1.0:1789760776.746201:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.746218:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6477, 6477] 00000008:00010000:1.0:1789760776.746236:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.746241:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.746253:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26529792, 26533888) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.746268:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26529792=0x194d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.746325:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.746327:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26533888 00000020:00010000:1.0:1789760776.746412:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.746416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.746420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.746424:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.746427:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.746435:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.746442:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.746458:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760776.746568:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 26529792<>26533888 00000100:00100000:0.0:1789760776.746624:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760776.746641:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495912064:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.778776:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973794 gen 2 00000008:00010000:0.0:1789760776.778833:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760776.778842:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.778860:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.778882:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760776.778917:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495912064:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.779082:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6477, 6477] 00000080:00200000:1.0:1789760776.779099:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.779105:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.779121:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26533888 i_blocks 53257 00000020:00200000:1.0:1789760776.779124:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.779134:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.779199:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26533888, count: 4096 00000080:00200000:1.0:1789760776.779207:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26533888, bytes: 4096 00000080:00200000:1.0:1789760776.779219:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.779228:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.779231:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26533888, 26537984) 0 0 00000080:00200000:1.0:1789760776.779236:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.779253:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6478, 6478] 00020000:00200000:1.0:1789760776.779261:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26533888 chunk: [26533888, 26537984) 26537984, 4096 00020000:00200000:1.0:1789760776.779266:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.779285:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6478, 6478] 00000008:00010000:1.0:1789760776.779303:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.779309:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.779532:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26533888, 26537984) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.779551:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26533888=0x194e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.779614:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.779617:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26537984 00000020:00010000:1.0:1789760776.779628:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.779631:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.779635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.779639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.779642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.779651:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.779659:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.779750:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760776.779945:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 26533888<>26537984 00000100:00100000:0.0:1789760776.780070:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760776.780091:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495912192:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.809993:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973795 gen 2 00000008:00010000:0.0:1789760776.810099:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760776.810108:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.810119:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.810142:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000080:00200000:1.0:1789760776.810315:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6478, 6478] 00000080:00200000:1.0:1789760776.810332:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.810336:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.810363:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26537984 i_blocks 53257 00000020:00200000:1.0:1789760776.810367:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000100:00100000:0.0:1789760776.810503:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495912192:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.810599:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.810635:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26537984, count: 4096 00000080:00200000:1.0:1789760776.810643:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26537984, bytes: 4096 00000080:00200000:1.0:1789760776.810653:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.810662:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.810664:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26537984, 26542080) 0 0 00000080:00200000:1.0:1789760776.810669:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.810687:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6479, 6479] 00020000:00200000:1.0:1789760776.810696:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26537984 chunk: [26537984, 26542080) 26542080, 4096 00020000:00200000:1.0:1789760776.810700:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.810720:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6479, 6479] 00000008:00010000:1.0:1789760776.810738:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.810744:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.810755:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26537984, 26542080) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.810770:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26537984=0x194f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.810828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.810831:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26542080 00000020:00010000:1.0:1789760776.810854:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.810858:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.810862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.810869:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.810872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.810880:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.810887:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.810904:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760776.811044:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 26537984<>26542080 00000100:00100000:1.0:1789760776.811092:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760776.811109:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495912320:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760776.851748:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760776.851765:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.851778:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.851804:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760776.851836:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495912320:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.852843:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6479, 6479] 00000080:00200000:1.0:1789760776.852863:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.852870:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.852886:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26542080 i_blocks 53257 00000020:00200000:1.0:1789760776.852891:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.852903:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.852939:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26542080, count: 4096 00000080:00200000:1.0:1789760776.852947:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26542080, bytes: 4096 00000080:00200000:1.0:1789760776.852960:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.852968:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.852971:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26542080, 26546176) 0 0 00000080:00200000:1.0:1789760776.852975:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.852993:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6480, 6480] 00020000:00200000:1.0:1789760776.853001:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26542080 chunk: [26542080, 26546176) 26546176, 4096 00020000:00200000:1.0:1789760776.853005:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.853025:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6480, 6480] 00000008:00010000:1.0:1789760776.853043:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.853050:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.853075:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26542080, 26546176) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.853091:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26542080=0x1950000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.853164:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.853209:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26546176 00000020:00010000:1.0:1789760776.853227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.853232:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.853235:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.853239:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.853242:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.853249:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.853256:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.853288:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760776.853431:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 26542080<>26546176 00000100:00100000:0.0:1789760776.853998:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760776.854025:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495912448:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.888528:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973797 gen 2 00000008:00010000:0.0:1789760776.888593:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760776.888602:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.888612:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.888635:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760776.888664:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495912448:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.888769:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6480, 6480] 00000080:00200000:1.0:1789760776.888787:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.888793:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.888809:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26546176 i_blocks 53257 00000020:00200000:1.0:1789760776.888813:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.888822:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.888856:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26546176, count: 4096 00000080:00200000:1.0:1789760776.888863:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26546176, bytes: 4096 00000080:00200000:1.0:1789760776.888873:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.888882:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.888884:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26546176, 26550272) 0 0 00000080:00200000:1.0:1789760776.888889:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.888907:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6481, 6481] 00020000:00200000:1.0:1789760776.888916:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26546176 chunk: [26546176, 26550272) 26550272, 4096 00020000:00200000:1.0:1789760776.888921:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.888942:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6481, 6481] 00000008:00010000:1.0:1789760776.888961:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.888966:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.888978:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26546176, 26550272) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.889084:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26546176=0x1951000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.889158:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.889161:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26550272 00000020:00010000:1.0:1789760776.889396:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.889401:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.889405:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.889409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.889412:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.889420:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.889427:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.889447:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760776.889586:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 26546176<>26550272 00000100:00100000:0.0:1789760776.889936:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760776.889960:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495912576:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.921719:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973798 gen 2 00000008:00010000:0.0:1789760776.922096:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760776.922208:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.922220:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.922244:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760776.922278:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495912576:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.922854:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6481, 6481] 00000080:00200000:1.0:1789760776.922874:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.922880:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.922897:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26550272 i_blocks 53257 00000020:00200000:1.0:1789760776.922902:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.922912:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.922947:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26550272, count: 4096 00000080:00200000:1.0:1789760776.922955:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26550272, bytes: 4096 00000080:00200000:1.0:1789760776.922969:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.922978:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.922980:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26550272, 26554368) 0 0 00000080:00200000:1.0:1789760776.922985:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.923003:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6482, 6482] 00020000:00200000:1.0:1789760776.923011:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26550272 chunk: [26550272, 26554368) 26554368, 4096 00020000:00200000:1.0:1789760776.923016:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.923037:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6482, 6482] 00000008:00010000:1.0:1789760776.923056:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.923062:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.923073:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26550272, 26554368) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.923089:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26550272=0x1952000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.923592:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.923596:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26554368 00000020:00010000:1.0:1789760776.923605:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.923609:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.923612:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.923616:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.923619:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.923628:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.923636:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.923657:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760776.923796:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 26550272<>26554368 00000100:00100000:0.0:1789760776.924029:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760776.924053:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495912704:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760776.966453:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760776.966469:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.966483:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.966505:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760776.966540:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495912704:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.966803:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6482, 6482] 00000080:00200000:1.0:1789760776.966820:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.966826:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.966842:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26554368 i_blocks 53257 00000020:00200000:1.0:1789760776.966845:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.966854:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.966892:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26554368, count: 4096 00000080:00200000:1.0:1789760776.966899:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26554368, bytes: 4096 00000080:00200000:1.0:1789760776.966908:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.966917:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.966920:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26554368, 26558464) 0 0 00000080:00200000:1.0:1789760776.966925:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.966943:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6483, 6483] 00020000:00200000:1.0:1789760776.966952:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26554368 chunk: [26554368, 26558464) 26558464, 4096 00020000:00200000:1.0:1789760776.966956:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.966977:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6483, 6483] 00000008:00010000:1.0:1789760776.966996:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.967001:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.967012:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26554368, 26558464) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.967027:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26554368=0x1953000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.967085:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.967089:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26558464 00000020:00010000:1.0:1789760776.967097:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.967101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.967104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.967107:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.967110:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.967117:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.967124:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.967140:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760776.967307:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 26554368<>26558464 00000100:00100000:0.0:1789760776.968435:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760776.968463:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495912832:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760776.997869:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973799 gen 2 00000100:00100000:0.0:1789760776.997910:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d500 x1876699495912832/t4294973800(4294973800) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760792 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760776.997961:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760776.997968:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.997979:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.997999:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760776.998025:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495912832:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760776.998203:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6483, 6483] 00000080:00200000:1.0:1789760776.998220:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.998225:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.998240:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26558464 i_blocks 53257 00000020:00200000:1.0:1789760776.998244:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.998254:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.998288:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26558464, count: 4096 00000080:00200000:1.0:1789760776.998296:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26558464, bytes: 4096 00000080:00200000:1.0:1789760776.998306:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.998315:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.998317:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26558464, 26562560) 0 0 00000080:00200000:1.0:1789760776.998332:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.998357:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6484, 6484] 00020000:00200000:1.0:1789760776.998366:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26558464 chunk: [26558464, 26562560) 26562560, 4096 00020000:00200000:1.0:1789760776.998371:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.998392:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6484, 6484] 00000008:00010000:1.0:1789760776.998412:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.998418:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760776.998429:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26558464, 26562560) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.998443:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26558464=0x1954000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.998498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.998501:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26562560 00000020:00010000:1.0:1789760776.998509:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.998512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.998516:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.998519:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.998523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.998529:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.998537:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.998554:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760776.998792:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 26558464<>26562560 00000100:00100000:0.0:1789760776.998953:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760776.998971:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495912960:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.031126:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973801 gen 2 00000008:00010000:0.0:1789760777.031217:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760777.031225:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.031235:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.031291:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760777.031318:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495912960:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.031447:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6484, 6484] 00000080:00200000:1.0:1789760777.031466:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.031473:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.031491:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26562560 i_blocks 53257 00000020:00200000:1.0:1789760777.031494:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.031503:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.031538:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26562560, count: 4096 00000080:00200000:1.0:1789760777.031545:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26562560, bytes: 4096 00000080:00200000:1.0:1789760777.031556:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.031565:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.031568:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26562560, 26566656) 0 0 00000080:00200000:1.0:1789760777.031572:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.031589:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6485, 6485] 00020000:00200000:1.0:1789760777.031597:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26562560 chunk: [26562560, 26566656) 26566656, 4096 00020000:00200000:1.0:1789760777.031602:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.031620:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6485, 6485] 00000008:00010000:1.0:1789760777.031636:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.031642:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.031653:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26562560, 26566656) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.031728:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26562560=0x1955000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.031780:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.031783:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26566656 00000020:00010000:1.0:1789760777.031791:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.031794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.031797:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.031800:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.031803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.031810:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.031816:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.031830:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760777.032125:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 26562560<>26566656 00000100:00100000:0.0:1789760777.032205:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495913088:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760777.062870:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760777.062889:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.062901:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.062921:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760777.062954:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495913088:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.063058:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6485, 6485] 00000080:00200000:1.0:1789760777.063075:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.063082:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.063097:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26566656 i_blocks 53257 00000020:00200000:1.0:1789760777.063101:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.063109:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.063145:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26566656, count: 4096 00000080:00200000:1.0:1789760777.063152:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26566656, bytes: 4096 00000080:00200000:1.0:1789760777.063162:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.063203:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.063205:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26566656, 26570752) 0 0 00000080:00200000:1.0:1789760777.063210:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.063228:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6486, 6486] 00020000:00200000:1.0:1789760777.063238:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26566656 chunk: [26566656, 26570752) 26570752, 4096 00020000:00200000:1.0:1789760777.063242:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.063265:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6486, 6486] 00000008:00010000:1.0:1789760777.063285:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.063291:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.063304:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26566656, 26570752) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.063505:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26566656=0x1956000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.063579:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.063582:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26570752 00000020:00010000:1.0:1789760777.063593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.063596:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.063599:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.063603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.063606:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.063614:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.063621:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.063641:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760777.064025:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 26566656<>26570752 00000100:00100000:0.0:1789760777.064109:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495913216:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.094983:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973803 gen 2 00000008:00010000:0.0:1789760777.095049:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760777.095058:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.095067:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.095088:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760777.095118:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495913216:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.095459:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6486, 6486] 00000080:00200000:1.0:1789760777.095476:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.095483:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.095500:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26570752 i_blocks 53257 00000020:00200000:1.0:1789760777.095504:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.095514:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.095549:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26570752, count: 4096 00000080:00200000:1.0:1789760777.095556:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26570752, bytes: 4096 00000080:00200000:1.0:1789760777.095566:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.095575:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.095577:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26570752, 26574848) 0 0 00000080:00200000:1.0:1789760777.095582:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.095600:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6487, 6487] 00020000:00200000:1.0:1789760777.095608:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26570752 chunk: [26570752, 26574848) 26574848, 4096 00020000:00200000:1.0:1789760777.095613:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.095633:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6487, 6487] 00000008:00010000:1.0:1789760777.095652:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.095658:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.095681:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26570752, 26574848) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.095906:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26570752=0x1957000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.095978:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.095982:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26574848 00000020:00010000:1.0:1789760777.095992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.095995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.095999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.096002:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.096005:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.096014:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.096021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.096038:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760777.096162:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 26570752<>26574848 00000100:00100000:0.0:1789760777.096827:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495913344:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.149679:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973804 gen 2 00000008:00010000:0.0:1789760777.149732:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760777.149741:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.149752:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.149829:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760777.149860:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495913344:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.152600:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6487, 6487] 00000080:00200000:1.0:1789760777.152642:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.152650:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.152671:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26574848 i_blocks 53257 00000020:00200000:1.0:1789760777.152678:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.152695:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.152736:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26574848, count: 4096 00000080:00200000:1.0:1789760777.152745:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26574848, bytes: 4096 00000080:00200000:1.0:1789760777.152761:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.152770:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.152773:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26574848, 26578944) 0 0 00000080:00200000:1.0:1789760777.152777:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.152795:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6488, 6488] 00020000:00200000:1.0:1789760777.152805:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26574848 chunk: [26574848, 26578944) 26578944, 4096 00020000:00200000:1.0:1789760777.152810:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.152833:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6488, 6488] 00000008:00010000:1.0:1789760777.152853:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.152860:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.152873:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26574848, 26578944) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.152890:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26574848=0x1958000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.153408:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.153413:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26578944 00000020:00010000:1.0:1789760777.153423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.153427:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.153501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.153505:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.153508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.153517:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.153525:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.153560:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760777.153765:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 26574848<>26578944 00000100:00100000:0.0:1789760777.153874:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495913472:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.212983:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973805 gen 2 00000008:00010000:0.0:1789760777.213034:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760777.213042:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.213054:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.213075:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760777.213102:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495913472:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.213260:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6488, 6488] 00000080:00200000:1.0:1789760777.213278:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.213284:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.213301:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26578944 i_blocks 53257 00000020:00200000:1.0:1789760777.213305:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.213314:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.213366:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26578944, count: 4096 00000080:00200000:1.0:1789760777.213383:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26578944, bytes: 4096 00000080:00200000:1.0:1789760777.213394:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.213402:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.213404:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26578944, 26583040) 0 0 00000080:00200000:1.0:1789760777.213409:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.213427:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6489, 6489] 00020000:00200000:1.0:1789760777.213436:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26578944 chunk: [26578944, 26583040) 26583040, 4096 00020000:00200000:1.0:1789760777.213440:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.213460:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6489, 6489] 00000008:00010000:1.0:1789760777.213477:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.213483:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.213495:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26578944, 26583040) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.213509:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26578944=0x1959000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.213554:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.213557:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26583040 00000020:00010000:1.0:1789760777.213563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.213566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.213568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.213570:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.213572:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.213577:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.213582:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.213594:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760777.213709:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 26578944<>26583040 00000100:00100000:0.0:1789760777.213781:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495913600:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760777.251137:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760777.251154:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.251166:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.251218:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760777.251248:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495913600:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.251335:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6489, 6489] 00000080:00200000:1.0:1789760777.251360:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.251367:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.251423:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26583040 i_blocks 53257 00000020:00200000:1.0:1789760777.251491:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.251502:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.251537:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26583040, count: 4096 00000080:00200000:1.0:1789760777.251545:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26583040, bytes: 4096 00000080:00200000:1.0:1789760777.251556:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.251564:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.251566:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26583040, 26587136) 0 0 00000080:00200000:1.0:1789760777.251571:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.251589:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6490, 6490] 00020000:00200000:1.0:1789760777.251598:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26583040 chunk: [26583040, 26587136) 26587136, 4096 00020000:00200000:1.0:1789760777.251603:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.251624:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6490, 6490] 00000008:00010000:1.0:1789760777.251642:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.251649:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.251661:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26583040, 26587136) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.251675:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26583040=0x195a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.251730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.251733:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26587136 00000020:00010000:1.0:1789760777.251742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.251745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.251748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0368/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.251752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.251755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.251761:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.251768:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.251784:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760777.252090:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 26583040<>26587136 00000100:00100000:0.0:1789760777.252189:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495913728:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.303522:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973807 gen 2 00000008:00010000:1.0:1789760777.303788:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:1.0:1789760777.303798:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.303808:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.303920:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:1.0:1789760777.303960:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495913728:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760777.304331:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6490, 6490] 00000080:00200000:0.0:1789760777.304355:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.304361:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.304449:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26587136 i_blocks 53257 00000020:00200000:1.0:1789760777.304453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.304462:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.304497:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26587136, count: 4096 00000080:00200000:1.0:1789760777.304503:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26587136, bytes: 4096 00000080:00200000:1.0:1789760777.304517:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.304525:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.304528:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26587136, 26591232) 0 0 00000080:00200000:1.0:1789760777.304532:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.304548:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6491, 6491] 00020000:00200000:1.0:1789760777.304557:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26587136 chunk: [26587136, 26591232) 26591232, 4096 00020000:00200000:1.0:1789760777.304562:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.304583:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6491, 6491] 00000008:00010000:1.0:1789760777.304600:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.304605:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.304678:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26587136, 26591232) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.304694:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26587136=0x195b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.304808:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.304811:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26591232 00000020:00010000:1.0:1789760777.304820:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.304823:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.304827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.304830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.304832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.304839:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.304846:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.304932:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760777.305053:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 26587136<>26591232 00000100:00100000:0.0:1789760777.305213:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495913856:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.353656:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973808 gen 2 00000008:00010000:0.0:1789760777.353710:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760777.353720:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.353731:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.353794:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760777.353825:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495913856:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.353911:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6491, 6491] 00000080:00200000:1.0:1789760777.353929:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.353936:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.353952:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26591232 i_blocks 53257 00000020:00200000:1.0:1789760777.353955:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.353966:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.354001:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26591232, count: 4096 00000080:00200000:1.0:1789760777.354008:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26591232, bytes: 4096 00000080:00200000:1.0:1789760777.354018:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.354027:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.354030:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26591232, 26595328) 0 0 00000080:00200000:1.0:1789760777.354035:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.354052:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6492, 6492] 00020000:00200000:1.0:1789760777.354060:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26591232 chunk: [26591232, 26595328) 26595328, 4096 00020000:00200000:1.0:1789760777.354065:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.354084:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6492, 6492] 00000008:00010000:1.0:1789760777.354103:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.354109:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.354122:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26591232, 26595328) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.354137:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26591232=0x195c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.354232:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.354235:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26595328 00000020:00010000:1.0:1789760777.354244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.354248:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.354251:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.354254:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.354257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.354264:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.354272:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.354290:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760777.354423:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 26591232<>26595328 00000100:00100000:0.0:1789760777.354494:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495913984:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.385219:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973809 gen 2 00000008:00010000:0.0:1789760777.385272:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760777.385282:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.385295:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.385326:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760777.385364:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495913984:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.385512:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6492, 6492] 00000080:00200000:1.0:1789760777.385529:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.385535:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.385551:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26595328 i_blocks 53257 00000020:00200000:1.0:1789760777.385555:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.385563:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.385594:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26595328, count: 4096 00000080:00200000:1.0:1789760777.385602:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26595328, bytes: 4096 00000080:00200000:1.0:1789760777.385612:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.385621:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.385623:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26595328, 26599424) 0 0 00000080:00200000:1.0:1789760777.385628:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.385646:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6493, 6493] 00020000:00200000:1.0:1789760777.385656:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26595328 chunk: [26595328, 26599424) 26599424, 4096 00020000:00200000:1.0:1789760777.385672:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.385693:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6493, 6493] 00000008:00010000:1.0:1789760777.385712:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.385718:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.385729:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26595328, 26599424) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.385744:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26595328=0x195d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.385798:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.385801:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26599424 00000020:00010000:1.0:1789760777.385810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.385813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.385816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.385819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.385822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.385829:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.385835:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.385851:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760777.386869:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 26595328<>26599424 00000100:00100000:0.0:1789760777.386949:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495914112:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.421735:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973810 gen 2 00000008:00010000:0.0:1789760777.421790:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760777.421800:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.421811:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.421833:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760777.421871:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495914112:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.422023:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6493, 6493] 00000080:00200000:1.0:1789760777.422041:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.422057:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.422073:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26599424 i_blocks 53257 00000020:00200000:1.0:1789760777.422077:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.422086:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.422121:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26599424, count: 4096 00000080:00200000:1.0:1789760777.422131:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26599424, bytes: 4096 00000080:00200000:1.0:1789760777.422140:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.422150:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.422152:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26599424, 26603520) 0 0 00000080:00200000:1.0:1789760777.422157:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.422211:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6494, 6494] 00020000:00200000:1.0:1789760777.422221:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26599424 chunk: [26599424, 26603520) 26603520, 4096 00020000:00200000:1.0:1789760777.422226:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.422244:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6494, 6494] 00000008:00010000:1.0:1789760777.422262:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.422267:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.422279:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26599424, 26603520) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.422295:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26599424=0x195e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.422356:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.422359:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26603520 00000020:00010000:1.0:1789760777.422368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.422371:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.422374:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.422378:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.422381:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.422388:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.422394:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.422649:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:0.0:1789760777.422782:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26599424<>26603520 00000100:00100000:0.0:1789760777.422813:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760777.422829:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495914240:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.461693:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973811 gen 2 00000008:00010000:0.0:1789760777.461743:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760777.461752:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.461763:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.461784:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760777.461814:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495914240:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.462017:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6494, 6494] 00000080:00200000:1.0:1789760777.462036:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.462041:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.462056:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26603520 i_blocks 53257 00000020:00200000:1.0:1789760777.462060:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.462069:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.462101:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26603520, count: 4096 00000080:00200000:1.0:1789760777.462108:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26603520, bytes: 4096 00000080:00200000:1.0:1789760777.462118:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.462127:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.462129:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26603520, 26607616) 0 0 00000080:00200000:1.0:1789760777.462134:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.462151:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6495, 6495] 00020000:00200000:1.0:1789760777.462159:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26603520 chunk: [26603520, 26607616) 26607616, 4096 00020000:00200000:1.0:1789760777.462163:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.462209:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6495, 6495] 00000008:00010000:1.0:1789760777.462228:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.462234:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.462246:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26603520, 26607616) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.462261:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26603520=0x195f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.462330:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.462334:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26607616 00000020:00010000:1.0:1789760777.462350:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.462354:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.462357:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.462361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.462364:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.462372:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.462381:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.462399:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760777.462525:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 26603520<>26607616 00000100:00100000:1.0:1789760777.462574:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760777.462593:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495914368:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.499241:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973812 gen 2 00000008:00010000:0.0:1789760777.499297:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760777.499305:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.499392:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.499455:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760777.499488:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495914368:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.499577:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6495, 6495] 00000080:00200000:1.0:1789760777.499595:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.499602:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.499619:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26607616 i_blocks 53257 00000020:00200000:1.0:1789760777.499623:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.499632:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.499837:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26607616, count: 4096 00000080:00200000:1.0:1789760777.499846:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26607616, bytes: 4096 00000080:00200000:1.0:1789760777.499978:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.499987:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.499989:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26607616, 26611712) 0 0 00000080:00200000:1.0:1789760777.500104:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.500123:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6496, 6496] 00020000:00200000:1.0:1789760777.500132:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26607616 chunk: [26607616, 26611712) 26611712, 4096 00020000:00200000:1.0:1789760777.500136:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.500157:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6496, 6496] 00000008:00010000:1.0:1789760777.500205:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.500212:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.500224:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26607616, 26611712) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.500240:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26607616=0x1960000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.500303:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.500306:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26611712 00000020:00010000:1.0:1789760777.500316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.500319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.500323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.500326:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.500329:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.500349:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.500357:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.500377:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760777.500506:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 26607616<>26611712 00000100:00100000:0.0:1789760777.500765:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495914496:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760777.551452:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760777.551469:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.551481:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.551505:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760777.551537:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495914496:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.551637:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6496, 6496] 00000080:00200000:1.0:1789760777.551654:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.551672:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.551689:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26611712 i_blocks 53257 00000020:00200000:1.0:1789760777.551693:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.551702:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.551737:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26611712, count: 4096 00000080:00200000:1.0:1789760777.551744:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26611712, bytes: 4096 00000080:00200000:1.0:1789760777.551755:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.551763:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.551766:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26611712, 26615808) 0 0 00000080:00200000:1.0:1789760777.551770:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.551788:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6497, 6497] 00020000:00200000:1.0:1789760777.551796:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26611712 chunk: [26611712, 26615808) 26615808, 4096 00020000:00200000:1.0:1789760777.551800:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.551820:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6497, 6497] 00000008:00010000:1.0:1789760777.551840:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.551845:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.551866:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26611712, 26615808) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.551881:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26611712=0x1961000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.551944:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.551947:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26615808 00000020:00010000:1.0:1789760777.551955:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.551959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.551962:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.551965:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.551968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.551975:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.551983:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.552000:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760777.552127:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 26611712<>26615808 00000100:00100000:0.0:1789760777.552378:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760777.552397:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495914624:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.576038:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973814 gen 2 00000008:00010000:0.0:1789760777.576112:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760777.576122:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.576132:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.576258:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760777.576288:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495914624:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.576478:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6497, 6497] 00000080:00200000:1.0:1789760777.576495:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.576501:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.576517:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26615808 i_blocks 53257 00000020:00200000:1.0:1789760777.576522:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.576532:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.576642:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26615808, count: 4096 00000080:00200000:1.0:1789760777.576650:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26615808, bytes: 4096 00000080:00200000:1.0:1789760777.576662:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.576671:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.576674:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26615808, 26619904) 0 0 00000080:00200000:1.0:1789760777.576679:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.576695:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6498, 6498] 00020000:00200000:1.0:1789760777.576704:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26615808 chunk: [26615808, 26619904) 26619904, 4096 00020000:00200000:1.0:1789760777.576709:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.576726:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6498, 6498] 00000008:00010000:1.0:1789760777.576743:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.576748:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.576758:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26615808, 26619904) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.576773:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26615808=0x1962000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.576824:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.576827:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26619904 00000020:00010000:1.0:1789760777.576926:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.576931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.576934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.576938:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.576940:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.576948:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.576955:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.576969:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760777.577087:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 26615808<>26619904 00000100:00100000:0.0:1789760777.577143:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495914752:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.604623:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973815 gen 2 00000008:00010000:0.0:1789760777.605047:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760777.605057:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.605069:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.605099:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760777.605136:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495914752:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.605423:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6498, 6498] 00000080:00200000:1.0:1789760777.605452:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.605460:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.605477:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26619904 i_blocks 53257 00000020:00200000:1.0:1789760777.605481:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.605492:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.605527:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26619904, count: 4096 00000080:00200000:1.0:1789760777.605536:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26619904, bytes: 4096 00000080:00200000:1.0:1789760777.605551:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.605560:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.605563:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26619904, 26624000) 0 0 00000080:00200000:1.0:1789760777.605567:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.605583:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6499, 6499] 00020000:00200000:1.0:1789760777.605592:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26619904 chunk: [26619904, 26624000) 26624000, 4096 00020000:00200000:1.0:1789760777.605597:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.605620:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6499, 6499] 00000008:00010000:1.0:1789760777.605641:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.605647:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.605663:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26619904, 26624000) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.605680:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26619904=0x1963000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.605737:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.605741:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26624000 00000020:00010000:1.0:1789760777.605749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.605753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.605757:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.605760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.605763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.605771:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.605778:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.605794:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760777.605925:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 26619904<>26624000 00000100:00100000:0.0:1789760777.606124:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495914880:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.651543:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973816 gen 2 00000008:00010000:0.0:1789760777.651716:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760777.651726:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.651736:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.651803:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760777.651833:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495914880:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.652494:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6499, 6499] 00000080:00200000:1.0:1789760777.652636:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.652642:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.652659:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26624000 i_blocks 53257 00000020:00200000:1.0:1789760777.652663:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.652675:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.652710:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26624000, count: 4096 00000080:00200000:1.0:1789760777.652774:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26624000, bytes: 4096 00000080:00200000:1.0:1789760777.652790:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.652799:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.652802:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26624000, 26628096) 0 0 00000080:00200000:1.0:1789760777.652807:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.652888:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6500, 6500] 00020000:00200000:1.0:1789760777.653602:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26624000 chunk: [26624000, 26628096) 26628096, 4096 00020000:00200000:1.0:1789760777.653611:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.653639:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6500, 6500] 00000008:00010000:1.0:1789760777.653660:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.653670:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.653684:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26624000, 26628096) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.653700:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26624000=0x1964000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.653955:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.653957:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26628096 00000020:00010000:1.0:1789760777.653965:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.653968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.653971:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.653973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.653975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.653985:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.653993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.654023:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760777.654157:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 26624000<>26628096 00000100:00100000:0.0:1789760777.654407:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760777.654433:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495915008:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760777.675489:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:0.0:1789760777.691918:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760777.691934:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.691945:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.692012:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760777.692057:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495915008:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.692220:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6500, 6500] 00000080:00200000:1.0:1789760777.692238:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.692246:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.692465:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26628096 i_blocks 53257 00000020:00200000:1.0:1789760777.692471:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.692481:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.692516:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26628096, count: 4096 00000080:00200000:1.0:1789760777.692524:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26628096, bytes: 4096 00000080:00200000:1.0:1789760777.692536:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.692546:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.692548:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26628096, 26632192) 0 0 00000080:00200000:1.0:1789760777.692553:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.692571:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6501, 6501] 00020000:00200000:1.0:1789760777.692580:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26628096 chunk: [26628096, 26632192) 26632192, 4096 00020000:00200000:1.0:1789760777.692585:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.692607:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6501, 6501] 00000008:00010000:1.0:1789760777.692626:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.692633:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.692645:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26628096, 26632192) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.692661:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26628096=0x1965000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.692814:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.692817:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26632192 00000020:00010000:1.0:1789760777.692827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.692830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.692834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.692838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.692842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.692851:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.692858:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.692878:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760777.693009:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 26628096<>26632192 00000100:00100000:0.0:1789760777.693205:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495915136:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.745142:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973817 gen 2 00000100:00100000:0.0:1789760777.745220:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d880 x1876699495915136/t4294973818(4294973818) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760793 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760777.745272:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760777.745280:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.745290:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.745727:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760777.745755:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495915136:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760777.745873:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6501, 6501] 00000080:00200000:2.0:1789760777.745890:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.745905:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.745921:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26632192 i_blocks 53257 00000020:00200000:2.0:1789760777.745925:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.745935:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.745969:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26632192, count: 4096 00000080:00200000:2.0:1789760777.745976:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26632192, bytes: 4096 00000080:00200000:2.0:1789760777.745988:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.745997:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.746000:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26632192, 26636288) 0 0 00000080:00200000:2.0:1789760777.746004:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.746020:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6502, 6502] 00020000:00200000:2.0:1789760777.746028:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26632192 chunk: [26632192, 26636288) 26636288, 4096 00020000:00200000:2.0:1789760777.746033:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.746058:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6502, 6502] 00000008:00010000:2.0:1789760777.746076:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.746083:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760777.746096:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26632192, 26636288) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.746111:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26632192=0x1966000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.746165:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.746199:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26636288 00000020:00010000:2.0:1789760777.746208:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.746212:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.746215:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.746218:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.746221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.746228:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.746236:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.746253:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760777.746756:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 26632192<>26636288 00000100:00100000:3.0:1789760777.746818:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.746888:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495915264:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760777.788709:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973819 gen 2 00000008:00010000:3.0:1789760777.788887:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760777.788897:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.788908:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.789015:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760777.789050:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495915264:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760777.789145:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6502, 6502] 00000080:00200000:1.0:1789760777.789164:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.789193:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.789210:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26636288 i_blocks 53257 00000020:00200000:1.0:1789760777.789214:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.789225:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.789259:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26636288, count: 4096 00000080:00200000:1.0:1789760777.789266:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26636288, bytes: 4096 00000080:00200000:1.0:1789760777.789281:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.789291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.789293:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26636288, 26640384) 0 0 00000080:00200000:1.0:1789760777.789298:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.789316:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6503, 6503] 00020000:00200000:1.0:1789760777.789325:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26636288 chunk: [26636288, 26640384) 26640384, 4096 00020000:00200000:1.0:1789760777.789330:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.789845:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6503, 6503] 00000008:00010000:1.0:1789760777.789868:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.789874:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760777.789888:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26636288, 26640384) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.789905:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26636288=0x1967000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.789968:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.789970:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26640384 00000020:00010000:1.0:1789760777.789979:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.789982:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.789985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.789989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.789992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.790001:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.790008:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.790029:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760777.790157:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 26636288<>26640384 00000100:00100000:1.0:1789760777.790468:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495915392:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760777.830527:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760777.830538:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.830548:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.830567:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760777.830593:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495915392:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760777.831214:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6503, 6503] 00000080:00200000:3.0:1789760777.831238:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.831244:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.831262:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26640384 i_blocks 53257 00000020:00200000:3.0:1789760777.831267:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.831277:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.831309:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26640384, count: 4096 00000080:00200000:3.0:1789760777.831316:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26640384, bytes: 4096 00000080:00200000:3.0:1789760777.831518:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.831530:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760777.831532:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26640384, 26644480) 0 0 00000080:00200000:3.0:1789760777.831537:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.831555:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6504, 6504] 00020000:00200000:3.0:1789760777.831564:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26640384 chunk: [26640384, 26644480) 26644480, 4096 00020000:00200000:3.0:1789760777.831568:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.831590:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6504, 6504] 00000008:00010000:3.0:1789760777.831608:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.831614:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760777.831626:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26640384, 26644480) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.831640:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26640384=0x1968000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.831706:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.831709:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26644480 00000020:00010000:3.0:1789760777.831717:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.831721:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.831724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.831728:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.831730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.831739:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.831746:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.831768:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760777.832121:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 26640384<>26644480 00000100:00100000:2.0:1789760777.832541:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760777.832567:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495915520:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760777.883369:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973821 gen 2 00000008:00010000:3.0:1789760777.883448:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:3.0:1789760777.883458:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.883469:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.883492:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:3.0:1789760777.883524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495915520:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760777.883526:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6504, 6504] 00000080:00200000:0.0:1789760777.883547:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.883554:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.883573:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26644480 i_blocks 53257 00000020:00200000:0.0:1789760777.883577:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.883588:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.883625:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26644480, count: 4096 00000080:00200000:0.0:1789760777.883632:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26644480, bytes: 4096 00000080:00200000:0.0:1789760777.883643:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.883652:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.883654:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26644480, 26648576) 0 0 00000080:00200000:0.0:1789760777.883659:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.883675:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6505, 6505] 00020000:00200000:0.0:1789760777.883683:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26644480 chunk: [26644480, 26648576) 26648576, 4096 00020000:00200000:0.0:1789760777.883687:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.883707:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6505, 6505] 00000008:00010000:0.0:1789760777.883725:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.883731:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760777.883743:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26644480, 26648576) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.883758:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26644480=0x1969000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.883831:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.883834:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26648576 00000020:00010000:0.0:1789760777.883852:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.883855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.883858:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.883862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.883865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.883873:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.883879:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.883894:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760777.884017:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26644480<>26648576 00000100:00100000:1.0:1789760777.884098:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495915648:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760777.952880:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973822 gen 2 00000008:00010000:0.0:1789760777.952940:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760777.952948:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.952959:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.952980:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760777.953008:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495915648:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760777.953588:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6505, 6505] 00000080:00200000:3.0:1789760777.953607:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.953613:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.953630:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26648576 i_blocks 53257 00000020:00200000:3.0:1789760777.953634:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.953644:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.953689:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26648576, count: 4096 00000080:00200000:3.0:1789760777.953697:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26648576, bytes: 4096 00000080:00200000:3.0:1789760777.953709:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.953717:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760777.953720:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26648576, 26652672) 0 0 00000080:00200000:3.0:1789760777.953724:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.953754:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6506, 6506] 00020000:00200000:3.0:1789760777.953762:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26648576 chunk: [26648576, 26652672) 26652672, 4096 00020000:00200000:3.0:1789760777.953766:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.953793:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6506, 6506] 00000008:00010000:3.0:1789760777.953813:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.953819:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760777.953832:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26648576, 26652672) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.953847:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26648576=0x196a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.953907:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.953910:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26652672 00000020:00010000:3.0:1789760777.953920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.953924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.953926:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.953930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.953933:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.953941:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.953948:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.953968:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760777.954093:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 26648576<>26652672 00000100:00100000:2.0:1789760777.955645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495915776:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.004359:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973823 gen 2 00000008:00010000:2.0:1789760778.004412:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760778.004420:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.004431:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.004532:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760778.004559:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495915776:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.004772:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6506, 6506] 00000080:00200000:3.0:1789760778.004791:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.004798:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.004814:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26652672 i_blocks 53257 00000020:00200000:3.0:1789760778.004818:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.004827:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.004861:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26652672, count: 4096 00000080:00200000:3.0:1789760778.004869:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26652672, bytes: 4096 00000080:00200000:3.0:1789760778.004882:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.004891:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.004893:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26652672, 26656768) 0 0 00000080:00200000:3.0:1789760778.004897:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.004912:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6507, 6507] 00020000:00200000:3.0:1789760778.004921:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26652672 chunk: [26652672, 26656768) 26656768, 4096 00020000:00200000:3.0:1789760778.004925:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.004946:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6507, 6507] 00000008:00010000:3.0:1789760778.004965:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.004972:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.004984:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26652672, 26656768) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.005000:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26652672=0x196b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.006656:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.006664:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26656768 00000020:00010000:3.0:1789760778.006678:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.006682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.006686:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.006690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.006693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.006703:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.006713:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.006726:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760778.006854:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 26652672<>26656768 00000100:00100000:2.0:1789760778.007629:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760778.007659:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495915904:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.046441:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973824 gen 2 00000008:00010000:2.0:1789760778.046492:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760778.046576:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.046590:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.046664:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760778.046702:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495915904:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.049007:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6507, 6507] 00000080:00200000:3.0:1789760778.049036:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.049042:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.049061:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26656768 i_blocks 53257 00000020:00200000:3.0:1789760778.049066:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.049082:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.049116:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26656768, count: 4096 00000080:00200000:3.0:1789760778.049126:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26656768, bytes: 4096 00000080:00200000:3.0:1789760778.049141:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.049151:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.049153:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26656768, 26660864) 0 0 00000080:00200000:3.0:1789760778.049158:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.049199:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6508, 6508] 00020000:00200000:3.0:1789760778.049208:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26656768 chunk: [26656768, 26660864) 26660864, 4096 00020000:00200000:3.0:1789760778.049213:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.049236:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6508, 6508] 00000008:00010000:3.0:1789760778.049255:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.049261:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.049273:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26656768, 26660864) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.049289:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26656768=0x196c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.049370:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.049373:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26660864 00000020:00010000:3.0:1789760778.049382:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.049385:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.049388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.049391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.049394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.049405:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.049413:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.049444:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760778.050207:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 26656768<>26660864 00000100:00100000:3.0:1789760778.050266:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.050291:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495916032:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760778.093432:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:3.0:1789760778.093448:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.093460:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.093538:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:3.0:1789760778.093570:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495916032:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760778.097031:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6508, 6508] 00000080:00200000:2.0:1789760778.097059:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.097066:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.097085:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26660864 i_blocks 53257 00000020:00200000:2.0:1789760778.097090:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.097106:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.097134:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26660864, count: 4096 00000080:00200000:2.0:1789760778.097142:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26660864, bytes: 4096 00000080:00200000:2.0:1789760778.097155:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.097164:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.097180:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26660864, 26664960) 0 0 00000080:00200000:2.0:1789760778.097185:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.097200:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6509, 6509] 00020000:00200000:2.0:1789760778.097208:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26660864 chunk: [26660864, 26664960) 26664960, 4096 00020000:00200000:2.0:1789760778.097212:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.097233:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6509, 6509] 00000008:00010000:2.0:1789760778.097252:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.097259:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760778.097272:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26660864, 26664960) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.097287:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26660864=0x196d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.097923:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.097928:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26664960 00000020:00010000:2.0:1789760778.097936:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.097940:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.097943:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.097947:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.097950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.097957:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.097963:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.097986:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760778.098116:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 26660864<>26664960 00000100:00100000:2.0:1789760778.098164:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760778.098215:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916160:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.131064:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973826 gen 2 00000008:00010000:2.0:1789760778.131130:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760778.131140:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.131151:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.131223:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760778.131259:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916160:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760778.132758:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6509, 6509] 00000080:00200000:0.0:1789760778.132785:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.132793:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.132812:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26664960 i_blocks 53257 00000020:00200000:0.0:1789760778.132817:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.132832:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.132880:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26664960, count: 4096 00000080:00200000:0.0:1789760778.132889:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26664960, bytes: 4096 00000080:00200000:0.0:1789760778.132902:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.132911:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.132914:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26664960, 26669056) 0 0 00000080:00200000:0.0:1789760778.132918:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.132935:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6510, 6510] 00020000:00200000:0.0:1789760778.132943:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26664960 chunk: [26664960, 26669056) 26669056, 4096 00020000:00200000:0.0:1789760778.132948:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.132969:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6510, 6510] 00000008:00010000:0.0:1789760778.132987:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.132992:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760778.133004:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26664960, 26669056) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.133017:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26664960=0x196e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.133082:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.133084:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26669056 00000020:00010000:0.0:1789760778.133092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.133096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.133099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.133102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.133105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.133112:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.133120:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.133142:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760778.133312:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26664960<>26669056 00000100:00100000:0.0:1789760778.133378:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760778.133403:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495916288:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760778.177158:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973827 gen 2 00000008:00010000:0.0:1789760778.177646:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760778.177656:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.177667:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.177765:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760778.177797:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495916288:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760778.177947:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6510, 6510] 00000080:00200000:2.0:1789760778.177965:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.177972:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.177987:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26669056 i_blocks 53257 00000020:00200000:2.0:1789760778.177991:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.178000:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.178039:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26669056, count: 4096 00000080:00200000:2.0:1789760778.178047:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26669056, bytes: 4096 00000080:00200000:2.0:1789760778.178060:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.178069:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.178072:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26669056, 26673152) 0 0 00000080:00200000:2.0:1789760778.178076:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.178093:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6511, 6511] 00020000:00200000:2.0:1789760778.178102:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26669056 chunk: [26669056, 26673152) 26673152, 4096 00020000:00200000:2.0:1789760778.178107:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.178128:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6511, 6511] 00000008:00010000:2.0:1789760778.178148:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.178154:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760778.178166:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26669056, 26673152) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.178922:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26669056=0x196f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.178993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.178996:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26673152 00000020:00010000:2.0:1789760778.179006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.179010:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.179013:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.179017:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.179019:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.179028:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.179036:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.179060:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760778.179380:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 26669056<>26673152 00000100:00100000:2.0:1789760778.179436:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916416:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.223969:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973828 gen 2 00000008:00010000:2.0:1789760778.224036:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760778.224045:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.224057:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.224081:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760778.224112:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916416:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.224722:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6511, 6511] 00000080:00200000:3.0:1789760778.224741:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.224748:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.224774:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26673152 i_blocks 53257 00000020:00200000:3.0:1789760778.224779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.224790:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.224824:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26673152, count: 4096 00000080:00200000:3.0:1789760778.224831:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26673152, bytes: 4096 00000080:00200000:3.0:1789760778.224843:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.224852:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.224854:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26673152, 26677248) 0 0 00000080:00200000:3.0:1789760778.224859:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.224877:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6512, 6512] 00020000:00200000:3.0:1789760778.224885:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26673152 chunk: [26673152, 26677248) 26677248, 4096 00020000:00200000:3.0:1789760778.224890:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.225379:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6512, 6512] 00000008:00010000:3.0:1789760778.225400:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.225406:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.225492:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26673152, 26677248) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.225508:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26673152=0x1970000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.225570:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.225573:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26677248 00000020:00010000:3.0:1789760778.225581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.225585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.225589:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.225593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.225596:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.225605:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.225613:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.225636:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760778.225761:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 26673152<>26677248 00000100:00100000:2.0:1789760778.226113:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760778.226138:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916544:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.268636:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973829 gen 2 00000008:00010000:2.0:1789760778.268759:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760778.268768:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.268780:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.268808:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:2.0:1789760778.268840:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916544:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.269245:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6512, 6512] 00000080:00200000:3.0:1789760778.269345:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.269352:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.269368:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26677248 i_blocks 53257 00000020:00200000:3.0:1789760778.269372:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.269381:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.269415:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26677248, count: 4096 00000080:00200000:3.0:1789760778.269423:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26677248, bytes: 4096 00000080:00200000:3.0:1789760778.269438:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.269448:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.269450:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26677248, 26681344) 0 0 00000080:00200000:3.0:1789760778.269455:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.269473:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6513, 6513] 00020000:00200000:3.0:1789760778.269481:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26677248 chunk: [26677248, 26681344) 26681344, 4096 00020000:00200000:3.0:1789760778.269486:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.269507:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6513, 6513] 00000008:00010000:3.0:1789760778.269527:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.269533:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.269546:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26677248, 26681344) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.269561:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26677248=0x1971000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.269622:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.269625:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26681344 00000020:00010000:3.0:1789760778.269635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.269638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.269642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.270797:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.270801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.270817:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.270827:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.270855:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760778.270989:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 26677248<>26681344 00000100:00100000:2.0:1789760778.271755:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916672:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.310085:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973830 gen 2 00000008:00010000:2.0:1789760778.310140:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:2.0:1789760778.310149:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.310160:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.310234:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:2.0:1789760778.310264:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916672:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760778.311553:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6513, 6513] 00000080:00200000:0.0:1789760778.311576:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.311584:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.311601:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26681344 i_blocks 53257 00000020:00200000:0.0:1789760778.311606:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.311619:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.311656:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26681344, count: 4096 00000080:00200000:0.0:1789760778.311663:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26681344, bytes: 4096 00000080:00200000:0.0:1789760778.311687:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.311697:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.311700:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26681344, 26685440) 0 0 00000080:00200000:0.0:1789760778.311705:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.311722:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6514, 6514] 00020000:00200000:0.0:1789760778.311736:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26681344 chunk: [26681344, 26685440) 26685440, 4096 00020000:00200000:0.0:1789760778.311740:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.311761:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6514, 6514] 00000008:00010000:0.0:1789760778.311782:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.311788:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760778.311800:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26681344, 26685440) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.311814:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26681344=0x1972000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.311878:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.311881:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26685440 00000020:00010000:0.0:1789760778.311889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.311894:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.311897:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.311900:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.311909:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.311918:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.311925:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.311946:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760778.312085:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26681344<>26685440 00000100:00100000:1.0:1789760778.312247:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760778.312275:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495916800:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760778.344981:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973831 gen 2 00000008:00010000:0.0:1789760778.345030:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760778.345039:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.345049:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.345076:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760778.345106:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495916800:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760778.345272:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6514, 6514] 00000080:00200000:2.0:1789760778.345290:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.345297:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.345313:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26685440 i_blocks 53257 00000020:00200000:2.0:1789760778.345317:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.345326:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.345380:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26685440, count: 4096 00000080:00200000:2.0:1789760778.345387:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26685440, bytes: 4096 00000080:00200000:2.0:1789760778.345396:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.345405:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.345408:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26685440, 26689536) 0 0 00000080:00200000:2.0:1789760778.345412:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.345430:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6515, 6515] 00020000:00200000:2.0:1789760778.345438:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26685440 chunk: [26685440, 26689536) 26689536, 4096 00020000:00200000:2.0:1789760778.345443:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.345727:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6515, 6515] 00000008:00010000:2.0:1789760778.345746:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.345753:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760778.345765:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26685440, 26689536) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.345780:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26685440=0x1973000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.345847:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.345851:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26689536 00000020:00010000:2.0:1789760778.345859:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.345863:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.345867:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.345870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.345874:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.345881:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.345888:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.345906:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760778.346032:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 26685440<>26689536 00000100:00100000:2.0:1789760778.346081:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760778.346301:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916928:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.374114:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973832 gen 2 00000008:00010000:2.0:1789760778.374208:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:2.0:1789760778.374217:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.374238:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.374302:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:2.0:1789760778.374352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495916928:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.374532:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6515, 6515] 00000080:00200000:3.0:1789760778.374550:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.374556:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.374665:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26689536 i_blocks 53257 00000020:00200000:2.0:1789760778.374671:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.374681:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.374720:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26689536, count: 4096 00000080:00200000:2.0:1789760778.374727:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26689536, bytes: 4096 00000080:00200000:2.0:1789760778.374739:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.374749:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.374752:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26689536, 26693632) 0 0 00000080:00200000:2.0:1789760778.374756:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.374774:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6516, 6516] 00020000:00200000:2.0:1789760778.374785:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26689536 chunk: [26689536, 26693632) 26693632, 4096 00020000:00200000:2.0:1789760778.374790:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.374884:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6516, 6516] 00000008:00010000:2.0:1789760778.374908:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.374915:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760778.375043:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26689536, 26693632) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.375061:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26689536=0x1974000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.375117:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.375120:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26693632 00000020:00010000:2.0:1789760778.375130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.375134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.375138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.375142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.375145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.375153:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.375161:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.375243:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760778.375442:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 26689536<>26693632 00000100:00100000:3.0:1789760778.375517:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917056:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760778.407483:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973833 gen 2 00000008:00010000:3.0:1789760778.407532:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760778.407541:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.407553:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.407580:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760778.407612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917056:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760778.407781:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6516, 6516] 00000080:00200000:2.0:1789760778.407798:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.407804:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.407819:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26693632 i_blocks 53257 00000020:00200000:2.0:1789760778.407823:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.407832:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.407930:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26693632, count: 4096 00000080:00200000:2.0:1789760778.407938:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26693632, bytes: 4096 00000080:00200000:2.0:1789760778.407949:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.407957:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.407960:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26693632, 26697728) 0 0 00000080:00200000:2.0:1789760778.407964:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.407980:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6517, 6517] 00020000:00200000:2.0:1789760778.407989:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26693632 chunk: [26693632, 26697728) 26697728, 4096 00020000:00200000:2.0:1789760778.407994:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.408013:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6517, 6517] 00000008:00010000:2.0:1789760778.408031:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.408037:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760778.408048:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26693632, 26697728) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.408062:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26693632=0x1975000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.408119:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.408122:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26697728 00000020:00010000:2.0:1789760778.408131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.408134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.408137:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.408140:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.408143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.408150:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.408157:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.408224:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760778.408804:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 26693632<>26697728 00000100:00100000:3.0:1789760778.408912:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.408996:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917184:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760778.455086:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760778.455103:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.455784:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.455858:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760778.455891:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917184:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760778.455970:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6517, 6517] 00000080:00200000:0.0:1789760778.455987:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.455993:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.456009:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26697728 i_blocks 53257 00000020:00200000:0.0:1789760778.456013:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.456022:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.456056:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26697728, count: 4096 00000080:00200000:0.0:1789760778.456063:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26697728, bytes: 4096 00000080:00200000:0.0:1789760778.456076:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.456085:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.456088:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26697728, 26701824) 0 0 00000080:00200000:0.0:1789760778.456093:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.456188:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6518, 6518] 00020000:00200000:0.0:1789760778.456199:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26697728 chunk: [26697728, 26701824) 26701824, 4096 00020000:00200000:0.0:1789760778.456204:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.456226:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6518, 6518] 00000008:00010000:0.0:1789760778.456244:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.456251:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760778.456262:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26697728, 26701824) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.456276:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26697728=0x1976000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.456415:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.456418:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26701824 00000020:00010000:0.0:1789760778.456426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.456430:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.456433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.456436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.456439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.456447:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.456454:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.456471:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760778.456596:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26697728<>26701824 00000100:00100000:1.0:1789760778.456670:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495917312:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760778.498746:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973834 gen 2 00000100:00100000:0.0:1789760778.498794:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211836680 x1876699495917312/t4294973835(4294973835) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760794 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760778.498851:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760778.498858:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.498868:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.498961:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760778.498986:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495917312:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.499108:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6518, 6518] 00000080:00200000:3.0:1789760778.499381:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.499387:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.499404:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26701824 i_blocks 53257 00000020:00200000:3.0:1789760778.499408:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.499417:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.499481:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26701824, count: 4096 00000080:00200000:3.0:1789760778.499489:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26701824, bytes: 4096 00000080:00200000:3.0:1789760778.499504:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.499512:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.499515:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26701824, 26705920) 0 0 00000080:00200000:3.0:1789760778.499519:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.499540:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6519, 6519] 00020000:00200000:3.0:1789760778.499557:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26701824 chunk: [26701824, 26705920) 26705920, 4096 00020000:00200000:3.0:1789760778.499562:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.499591:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6519, 6519] 00000008:00010000:3.0:1789760778.499614:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.499621:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.499633:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26701824, 26705920) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.499649:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26701824=0x1977000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.499716:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.499719:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26705920 00000020:00010000:3.0:1789760778.499730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.499734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.499737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.499740:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.499744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.499751:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.499759:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.499780:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760778.499925:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 26701824<>26705920 00000100:00100000:2.0:1789760778.500511:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917440:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.546748:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973836 gen 2 00000008:00010000:2.0:1789760778.546816:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760778.546824:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.546834:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.546857:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760778.546887:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917440:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.546993:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6519, 6519] 00000080:00200000:3.0:1789760778.547010:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.547016:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.547033:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26705920 i_blocks 53257 00000020:00200000:3.0:1789760778.547036:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.547046:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.547083:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26705920, count: 4096 00000080:00200000:3.0:1789760778.547090:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26705920, bytes: 4096 00000080:00200000:3.0:1789760778.547100:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.547110:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.547127:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26705920, 26710016) 0 0 00000080:00200000:3.0:1789760778.547132:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.547151:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6520, 6520] 00020000:00200000:3.0:1789760778.547160:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26705920 chunk: [26705920, 26710016) 26710016, 4096 00020000:00200000:3.0:1789760778.547166:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.547238:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6520, 6520] 00000008:00010000:3.0:1789760778.547257:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.547264:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.547275:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26705920, 26710016) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.547290:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26705920=0x1978000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.547361:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.547365:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26710016 00000020:00010000:3.0:1789760778.547372:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.547376:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.547379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.547382:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.547385:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.547393:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.547400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.547417:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760778.548625:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 26705920<>26710016 00000100:00100000:3.0:1789760778.548697:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.548720:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917568:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760778.625817:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973837 gen 2 00000008:00010000:3.0:1789760778.625926:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:3.0:1789760778.625934:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.625947:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.625969:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:3.0:1789760778.625999:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917568:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760778.626073:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6520, 6520] 00000080:00200000:2.0:1789760778.626090:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.626097:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.626116:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26710016 i_blocks 53257 00000020:00200000:2.0:1789760778.626121:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.626132:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.626201:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26710016, count: 4096 00000080:00200000:2.0:1789760778.626208:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26710016, bytes: 4096 00000080:00200000:2.0:1789760778.626223:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.626245:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.626247:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26710016, 26714112) 0 0 00000080:00200000:2.0:1789760778.626251:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.626268:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6521, 6521] 00020000:00200000:2.0:1789760778.626276:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26710016 chunk: [26710016, 26714112) 26714112, 4096 00020000:00200000:2.0:1789760778.626280:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.626301:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6521, 6521] 00000008:00010000:2.0:1789760778.626319:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.626341:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760778.626354:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26710016, 26714112) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.626368:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26710016=0x1979000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.626580:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.626584:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26714112 00000020:00010000:2.0:1789760778.626592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.626596:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.626599:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.626877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.626881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.627523:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.627534:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.627557:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760778.627748:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 26710016<>26714112 00000100:00100000:2.0:1789760778.627806:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760778.627827:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495917696:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.670725:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973838 gen 2 00000008:00010000:2.0:1789760778.670781:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760778.670789:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.670801:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.670819:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760778.670849:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495917696:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.672161:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6521, 6521] 00000080:00200000:3.0:1789760778.672226:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.672234:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.672252:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26714112 i_blocks 53257 00000020:00200000:3.0:1789760778.672258:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.672269:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.672306:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26714112, count: 4096 00000080:00200000:3.0:1789760778.672315:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26714112, bytes: 4096 00000080:00200000:3.0:1789760778.672326:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.672345:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.672347:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26714112, 26718208) 0 0 00000080:00200000:3.0:1789760778.672352:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.672368:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6522, 6522] 00020000:00200000:3.0:1789760778.672376:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26714112 chunk: [26714112, 26718208) 26718208, 4096 00020000:00200000:3.0:1789760778.672381:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.672403:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6522, 6522] 00000008:00010000:3.0:1789760778.672422:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.672428:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.672440:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26714112, 26718208) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.672467:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26714112=0x197a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.672533:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.672536:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26718208 00000020:00010000:3.0:1789760778.672544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.672548:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.672551:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.672556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.672559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.672567:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.672574:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.672594:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760778.672740:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 26714112<>26718208 00000100:00100000:3.0:1789760778.672794:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.672818:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760778.704632:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973839 gen 2 00000008:00010000:3.0:1789760778.704747:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:3.0:1789760778.704755:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.704767:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.705011:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:3.0:1789760778.705051:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495917824:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760778.705137:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6522, 6522] 00000080:00200000:0.0:1789760778.705152:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.705159:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.705194:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26718208 i_blocks 53257 00000020:00200000:0.0:1789760778.705197:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.705207:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.705244:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26718208, count: 4096 00000080:00200000:0.0:1789760778.705251:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26718208, bytes: 4096 00000080:00200000:0.0:1789760778.705261:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.705270:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.705273:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26718208, 26722304) 0 0 00000080:00200000:0.0:1789760778.705277:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.705293:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6523, 6523] 00020000:00200000:0.0:1789760778.705302:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26718208 chunk: [26718208, 26722304) 26722304, 4096 00020000:00200000:0.0:1789760778.705307:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.705327:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6523, 6523] 00000008:00010000:0.0:1789760778.705357:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.705363:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760778.705374:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26718208, 26722304) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.705387:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26718208=0x197b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.705437:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.705440:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26722304 00000020:00010000:0.0:1789760778.705447:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.705451:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.705453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.705457:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.705460:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.705467:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.705474:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.705489:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760778.705743:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 26718208<>26722304 00000100:00100000:1.0:1789760778.705894:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495917952:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760778.706468:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760778.764610:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973840 gen 2 00000008:00010000:0.0:1789760778.764819:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760778.764828:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.764840:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.764867:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000080:00200000:3.0:1789760778.764989:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6523, 6523] 00000080:00200000:3.0:1789760778.765008:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.765015:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.765030:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26722304 i_blocks 53257 00000020:00200000:3.0:1789760778.765034:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.765043:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.765080:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26722304, count: 4096 00000080:00200000:3.0:1789760778.765088:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26722304, bytes: 4096 00000080:00200000:3.0:1789760778.765101:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.765111:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.765113:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26722304, 26726400) 0 0 00000080:00200000:3.0:1789760778.765118:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.765136:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6524, 6524] 00020000:00200000:3.0:1789760778.765144:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26722304 chunk: [26722304, 26726400) 26726400, 4096 00020000:00200000:3.0:1789760778.765149:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.765210:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6524, 6524] 00000008:00010000:3.0:1789760778.765230:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.765235:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.765247:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26722304, 26726400) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.765262:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26722304=0x197c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.765311:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.765314:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26726400 00000020:00010000:3.0:1789760778.765323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.765326:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.765340:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.765344:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.765346:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.765353:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.765360:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1789760778.765525:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495917952:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760778.765574:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:0.0:1789760778.765778:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 26722304<>26726400 00000100:00100000:0.0:1789760778.765820:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495918080:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760778.802074:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973841 gen 2 00000008:00010000:0.0:1789760778.802233:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:0.0:1789760778.802243:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.802255:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.802414:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:0.0:1789760778.802466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495918080:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.802781:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6524, 6524] 00000080:00200000:3.0:1789760778.802799:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.802805:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.802821:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26726400 i_blocks 53257 00000020:00200000:3.0:1789760778.802825:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.802836:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.802871:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26726400, count: 4096 00000080:00200000:3.0:1789760778.802878:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26726400, bytes: 4096 00000080:00200000:3.0:1789760778.803009:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.803017:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.803020:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26726400, 26730496) 0 0 00000080:00200000:3.0:1789760778.803024:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.803042:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6525, 6525] 00020000:00200000:3.0:1789760778.803050:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26726400 chunk: [26726400, 26730496) 26730496, 4096 00020000:00200000:3.0:1789760778.803055:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.803075:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6525, 6525] 00000008:00010000:3.0:1789760778.803093:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.803099:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760778.804253:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26726400, 26730496) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.804279:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26726400=0x197d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.804370:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.804374:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26730496 00000020:00010000:3.0:1789760778.804385:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.804388:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.804392:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.804396:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.804399:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.804411:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.804649:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.804720:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760778.804863:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 26726400<>26730496 00000100:00100000:2.0:1789760778.805236:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495918208:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.847943:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973842 gen 2 00000008:00010000:2.0:1789760778.847998:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760778.848008:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.848019:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.848125:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760778.848233:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495918208:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.848374:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6525, 6525] 00000080:00200000:3.0:1789760778.848394:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.848402:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.848577:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26730496 i_blocks 53257 00000020:00200000:2.0:1789760778.848582:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.848593:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.848630:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26730496, count: 4096 00000080:00200000:2.0:1789760778.848638:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26730496, bytes: 4096 00000080:00200000:2.0:1789760778.848714:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.848724:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.848727:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26730496, 26734592) 0 0 00000080:00200000:2.0:1789760778.848731:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.848748:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6526, 6526] 00020000:00200000:2.0:1789760778.848756:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26730496 chunk: [26730496, 26734592) 26734592, 4096 00020000:00200000:2.0:1789760778.848761:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.848782:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6526, 6526] 00000008:00010000:2.0:1789760778.848801:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.848806:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760778.848818:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26730496, 26734592) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.848832:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26730496=0x197e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.848947:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.848950:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26734592 00000020:00010000:2.0:1789760778.848959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.848963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.848966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.848970:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.848973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.848980:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.848987:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.849039:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760778.849165:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 26730496<>26734592 00000100:00100000:2.0:1789760778.849240:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760778.849260:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495918336:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760778.897791:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973843 gen 2 00000008:00010000:2.0:1789760778.897843:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760778.897899:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.897912:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.898011:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760778.898040:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495918336:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760778.898513:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6526, 6526] 00000080:00200000:3.0:1789760778.898928:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.898936:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.901039:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26734592 i_blocks 53257 00000020:00200000:2.0:1789760778.901053:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.901071:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.901109:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26734592, count: 4096 00000080:00200000:2.0:1789760778.901391:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26734592, bytes: 4096 00000080:00200000:2.0:1789760778.901407:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.901416:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.901419:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26734592, 26738688) 0 0 00000080:00200000:2.0:1789760778.901424:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.901443:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6527, 6527] 00020000:00200000:2.0:1789760778.901453:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26734592 chunk: [26734592, 26738688) 26738688, 4096 00020000:00200000:2.0:1789760778.901457:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.901480:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6527, 6527] 00000008:00010000:2.0:1789760778.901499:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.901506:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760778.901518:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26734592, 26738688) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.901533:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26734592=0x197f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.901606:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.901609:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26738688 00000020:00010000:2.0:1789760778.901617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.901621:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.901624:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.901628:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.901632:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.901641:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.901648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.901706:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760778.901832:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 26734592<>26738688 00000100:00100000:3.0:1789760778.902020:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495918464:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760778.936317:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760778.936345:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.936358:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.936373:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760778.936405:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495918464:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760778.936952:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6527, 6527] 00000080:00200000:0.0:1789760778.936970:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.936978:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.936994:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26738688 i_blocks 53257 00000020:00200000:0.0:1789760778.936998:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.937008:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.937045:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26738688, count: 4096 00000080:00200000:0.0:1789760778.937053:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26738688, bytes: 4096 00000080:00200000:0.0:1789760778.937067:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.937076:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.937079:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26738688, 26742784) 0 0 00000080:00200000:0.0:1789760778.937083:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.937101:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6528, 6528] 00020000:00200000:0.0:1789760778.937111:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26738688 chunk: [26738688, 26742784) 26742784, 4096 00020000:00200000:0.0:1789760778.937223:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.937247:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6528, 6528] 00000008:00010000:0.0:1789760778.937268:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.937274:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760778.937286:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26738688, 26742784) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.937302:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26738688=0x1980000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.937370:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.937373:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26742784 00000020:00010000:0.0:1789760778.937382:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.937385:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.937389:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.937392:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.937395:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.937403:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.937411:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.937430:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760778.937565:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 26738688<>26742784 00000100:00100000:1.0:1789760778.937647:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495918592:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760778.972368:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973844 gen 2 00000100:00100000:0.0:1789760778.972410:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211836a00 x1876699495918592/t4294973845(4294973845) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760794 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760778.972537:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760778.972659:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.972671:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.972694:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760778.972778:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495918592:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760778.972799:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6528, 6528] 00000080:00200000:1.0:1789760778.972816:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.972822:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.972838:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26742784 i_blocks 53257 00000020:00200000:1.0:1789760778.972842:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.972851:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.972882:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26742784, count: 4096 00000080:00200000:1.0:1789760778.972888:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26742784, bytes: 4096 00000080:00200000:1.0:1789760778.972898:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.972905:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.972907:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26742784, 26746880) 0 0 00000080:00200000:1.0:1789760778.972910:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.972925:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6529, 6529] 00020000:00200000:1.0:1789760778.972932:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26742784 chunk: [26742784, 26746880) 26746880, 4096 00020000:00200000:1.0:1789760778.972935:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.972954:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6529, 6529] 00000008:00010000:1.0:1789760778.972969:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.972973:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760778.972983:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26742784, 26746880) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.972995:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26742784=0x1981000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.973048:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.973051:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26746880 00000020:00010000:1.0:1789760778.973059:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.973063:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.973066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.973070:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.973073:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.973081:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.973087:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.973099:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760778.973317:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 26742784<>26746880 00000100:00100000:0.0:1789760778.973777:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495918720:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760779.012382:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973846 gen 2 00000008:00010000:0.0:1789760779.012454:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760779.012464:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.012474:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.012507:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760779.012541:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495918720:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.012888:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6529, 6529] 00000080:00200000:2.0:1789760779.012906:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.012912:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.012929:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26746880 i_blocks 53257 00000020:00200000:2.0:1789760779.012933:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.012942:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.012977:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26746880, count: 4096 00000080:00200000:2.0:1789760779.012986:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26746880, bytes: 4096 00000080:00200000:2.0:1789760779.013000:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.013008:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.013010:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26746880, 26750976) 0 0 00000080:00200000:2.0:1789760779.013014:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.013032:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6530, 6530] 00020000:00200000:2.0:1789760779.013041:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26746880 chunk: [26746880, 26750976) 26750976, 4096 00020000:00200000:2.0:1789760779.013045:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.013065:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6530, 6530] 00000008:00010000:2.0:1789760779.013085:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.013091:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.013102:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26746880, 26750976) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.013119:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26746880=0x1982000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.013334:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.013338:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26750976 00000020:00010000:2.0:1789760779.013346:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.013350:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.013353:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.013357:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.013360:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.013367:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.013373:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.014482:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:0.0:1789760779.014638:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 26746880<>26750976 00000100:00100000:1.0:1789760779.014921:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760779.022796:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495918848:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760779.055028:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:0.0:1789760779.055045:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.055058:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.055158:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:0.0:1789760779.055239:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495918848:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.055478:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6530, 6530] 00000080:00200000:2.0:1789760779.055496:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.055502:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.055601:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26750976 i_blocks 53257 00000020:00200000:2.0:1789760779.055605:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.055615:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.055715:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26750976, count: 4096 00000080:00200000:2.0:1789760779.055723:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26750976, bytes: 4096 00000080:00200000:2.0:1789760779.055737:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.055746:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.055748:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26750976, 26755072) 0 0 00000080:00200000:2.0:1789760779.055753:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.055769:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6531, 6531] 00020000:00200000:2.0:1789760779.055776:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26750976 chunk: [26750976, 26755072) 26755072, 4096 00020000:00200000:2.0:1789760779.055781:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.055803:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6531, 6531] 00000008:00010000:2.0:1789760779.055822:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.055827:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.055839:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26750976, 26755072) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.055854:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26750976=0x1983000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.055908:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.055911:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26755072 00000020:00010000:2.0:1789760779.055920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.055923:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.055926:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.055930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.055932:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.055939:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.055945:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.055958:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760779.056129:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 26750976<>26755072 00000100:00100000:3.0:1789760779.056218:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495918976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.094220:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973848 gen 2 00000008:00010000:3.0:1789760779.094292:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760779.094303:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.094597:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.094741:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760779.094775:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495918976:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760779.094990:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6531, 6531] 00000080:00200000:1.0:1789760779.095010:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.095019:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.095037:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26755072 i_blocks 53257 00000020:00200000:1.0:1789760779.095041:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.095051:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.095090:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26755072, count: 4096 00000080:00200000:1.0:1789760779.095098:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26755072, bytes: 4096 00000080:00200000:1.0:1789760779.095112:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.095122:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.095124:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26755072, 26759168) 0 0 00000080:00200000:1.0:1789760779.095129:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.095146:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6532, 6532] 00020000:00200000:1.0:1789760779.095155:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26755072 chunk: [26755072, 26759168) 26759168, 4096 00020000:00200000:1.0:1789760779.095159:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.095227:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6532, 6532] 00000008:00010000:1.0:1789760779.095246:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.095252:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b330, osc lock ffff8dd205030130, flags 0x0 00000080:00200000:1.0:1789760779.095264:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26755072, 26759168) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.095278:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26755072=0x1984000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.096225:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.096230:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26759168 00000020:00010000:0.0:1789760779.096240:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.096243:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.096246:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b330 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.096248:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.096251:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b330 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.096259:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.096265:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.096278:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760779.096495:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 26755072<>26759168 00000100:00100000:0.0:1789760779.096552:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760779.096576:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495919104:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760779.136248:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973849 gen 2 00000008:00010000:0.0:1789760779.136303:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:0.0:1789760779.136426:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.136505:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.136575:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:0.0:1789760779.136606:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495919104:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760779.136889:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6532, 6532] 00000080:00200000:1.0:1789760779.136907:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.136919:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.136935:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26759168 i_blocks 53257 00000020:00200000:1.0:1789760779.136938:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.136946:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.136980:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26759168, count: 4096 00000080:00200000:1.0:1789760779.136986:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26759168, bytes: 4096 00000080:00200000:1.0:1789760779.136994:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.137001:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.137003:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26759168, 26763264) 0 0 00000080:00200000:1.0:1789760779.137007:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.137021:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6533, 6533] 00020000:00200000:1.0:1789760779.137029:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26759168 chunk: [26759168, 26763264) 26763264, 4096 00020000:00200000:1.0:1789760779.137034:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.137207:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6533, 6533] 00000008:00010000:1.0:1789760779.137229:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.137235:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760779.137247:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26759168, 26763264) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.137262:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26759168=0x1985000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.137502:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.137506:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26763264 00000020:00010000:1.0:1789760779.137515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.137519:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.137522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.137526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.137529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.137538:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.137545:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.137565:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760779.137772:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 26759168<>26763264 00000100:00100000:0.0:1789760779.137881:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495919232:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760779.151545:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495919488:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760779.151694:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1789760779.151703:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495919360:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760779.151741:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495919616:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760779.157993:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2356:1876699495919360:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760779.159462:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495919616:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760779.159481:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495919488:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760779.176971:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973850 gen 2 00000008:00010000:1.0:1789760779.177024:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760779.177033:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.177045:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.177070:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760779.177100:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495919232:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760779.177208:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6533, 6533] 00000080:00200000:0.0:1789760779.177227:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.177234:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.177251:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26763264 i_blocks 53257 00000020:00200000:0.0:1789760779.177254:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.177264:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.177297:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26763264, count: 4096 00000080:00200000:0.0:1789760779.177304:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26763264, bytes: 4096 00000080:00200000:0.0:1789760779.177322:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.177344:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.177346:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26763264, 26767360) 0 0 00000080:00200000:0.0:1789760779.177351:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.177368:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6534, 6534] 00020000:00200000:0.0:1789760779.177375:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26763264 chunk: [26763264, 26767360) 26767360, 4096 00020000:00200000:0.0:1789760779.177380:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.177400:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6534, 6534] 00000008:00010000:0.0:1789760779.177418:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.177424:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760779.177436:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26763264, 26767360) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.177452:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26763264=0x1986000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.177516:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.177519:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26767360 00000020:00010000:0.0:1789760779.177527:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.177531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.177534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.177538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.177541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.177548:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.177557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.177573:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760779.177707:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 26763264<>26767360 00000100:00100000:1.0:1789760779.177871:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760779.177889:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495919744:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760779.215856:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973851 gen 2 00000008:00010000:1.0:1789760779.215909:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760779.215918:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.215928:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.215947:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760779.215987:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495919744:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760779.216207:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6534, 6534] 00000080:00200000:0.0:1789760779.216225:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.216232:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.216256:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26767360 i_blocks 53257 00000020:00200000:0.0:1789760779.216260:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.216269:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.216303:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26767360, count: 4096 00000080:00200000:0.0:1789760779.216310:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26767360, bytes: 4096 00000080:00200000:0.0:1789760779.216319:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.216338:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.216341:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26767360, 26771456) 0 0 00000080:00200000:0.0:1789760779.216346:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.216363:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6535, 6535] 00020000:00200000:0.0:1789760779.216371:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26767360 chunk: [26767360, 26771456) 26771456, 4096 00020000:00200000:0.0:1789760779.216375:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.216395:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6535, 6535] 00000008:00010000:0.0:1789760779.216413:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.216418:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760779.216430:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26767360, 26771456) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.216444:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26767360=0x1987000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.216500:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.216504:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26771456 00000020:00010000:0.0:1789760779.216512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.216516:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.216519:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.216522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.216526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.216533:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.216540:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.216556:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760779.216681:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 26767360<>26771456 00000100:00100000:1.0:1789760779.216847:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495919872:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760779.262261:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973852 gen 2 00000008:00010000:1.0:1789760779.262464:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760779.262472:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.262483:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.262505:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000080:00200000:0.0:1789760779.262529:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6535, 6535] 00000100:00100000:1.0:1789760779.262534:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495919872:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760779.262548:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.262552:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.262568:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26771456 i_blocks 53257 00000020:00200000:0.0:1789760779.262571:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.262580:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.262615:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26771456, count: 4096 00000080:00200000:0.0:1789760779.262621:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26771456, bytes: 4096 00000080:00200000:0.0:1789760779.262631:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.262639:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.262641:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26771456, 26775552) 0 0 00000080:00200000:0.0:1789760779.262646:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.262663:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6536, 6536] 00020000:00200000:0.0:1789760779.262670:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26771456 chunk: [26771456, 26775552) 26775552, 4096 00020000:00200000:0.0:1789760779.262675:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.262700:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6536, 6536] 00000008:00010000:0.0:1789760779.262717:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.262722:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760779.262734:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26771456, 26775552) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.262745:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26771456=0x1988000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.262800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.262802:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26775552 00000020:00010000:0.0:1789760779.262809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.262811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.262813:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.262816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.262818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.262823:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.262828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.262840:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760779.263034:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 26771456<>26775552 00000100:00100000:1.0:1789760779.263139:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760779.263154:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495920000:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760779.303419:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760779.303444:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.303456:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.303521:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760779.303554:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495920000:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760779.305846:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6536, 6536] 00000080:00200000:0.0:1789760779.305875:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.305882:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.305901:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26775552 i_blocks 53257 00000020:00200000:0.0:1789760779.305906:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.305920:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.305957:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26775552, count: 4096 00000080:00200000:0.0:1789760779.305966:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26775552, bytes: 4096 00000080:00200000:0.0:1789760779.305977:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.305987:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.305989:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26775552, 26779648) 0 0 00000080:00200000:0.0:1789760779.305994:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.306012:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6537, 6537] 00020000:00200000:0.0:1789760779.306021:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26775552 chunk: [26775552, 26779648) 26779648, 4096 00020000:00200000:0.0:1789760779.306026:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.306048:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6537, 6537] 00000008:00010000:0.0:1789760779.306067:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.306074:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760779.306086:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26775552, 26779648) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.306099:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26775552=0x1989000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.306199:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.306203:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26779648 00000020:00010000:0.0:1789760779.306212:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.306216:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.306219:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.306224:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.306227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.306235:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.306242:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.306275:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760779.306874:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 26775552<>26779648 00000100:00100000:1.0:1789760779.306948:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495920128:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760779.348436:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973854 gen 2 00000008:00010000:1.0:1789760779.348504:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760779.348511:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.348527:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.348551:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760779.348576:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495920128:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.348670:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6537, 6537] 00000080:00200000:2.0:1789760779.348685:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.348693:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.348708:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26779648 i_blocks 53257 00000020:00200000:2.0:1789760779.348712:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.348722:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.348756:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26779648, count: 4096 00000080:00200000:2.0:1789760779.348767:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26779648, bytes: 4096 00000080:00200000:2.0:1789760779.348777:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.348784:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.348786:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26779648, 26783744) 0 0 00000080:00200000:2.0:1789760779.348791:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.348809:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6538, 6538] 00020000:00200000:2.0:1789760779.348818:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26779648 chunk: [26779648, 26783744) 26783744, 4096 00020000:00200000:2.0:1789760779.348822:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.348843:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6538, 6538] 00000008:00010000:2.0:1789760779.348862:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.348868:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.348880:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26779648, 26783744) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.348894:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26779648=0x198a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.348952:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.348956:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26783744 00000020:00010000:2.0:1789760779.348964:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.348968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.348971:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.348974:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.348976:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.348983:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.348991:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.349007:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760779.349131:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 26779648<>26783744 00000100:00100000:3.0:1789760779.349264:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.349551:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920256:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.388643:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973855 gen 2 00000008:00010000:3.0:1789760779.388695:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760779.388704:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.388849:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.388873:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760779.388903:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920256:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.389939:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6538, 6538] 00000080:00200000:2.0:1789760779.389959:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.389965:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.389983:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26783744 i_blocks 53257 00000020:00200000:2.0:1789760779.389987:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.389997:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.390033:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26783744, count: 4096 00000080:00200000:2.0:1789760779.390041:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26783744, bytes: 4096 00000080:00200000:2.0:1789760779.390054:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.390063:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.390065:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26783744, 26787840) 0 0 00000080:00200000:2.0:1789760779.390070:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.390395:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6539, 6539] 00020000:00200000:2.0:1789760779.390404:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26783744 chunk: [26783744, 26787840) 26787840, 4096 00020000:00200000:2.0:1789760779.390409:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.390431:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6539, 6539] 00000008:00010000:2.0:1789760779.390451:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.390457:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.390469:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26783744, 26787840) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.390487:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26783744=0x198b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.390544:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.390547:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26787840 00000020:00010000:2.0:1789760779.390555:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.390559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.390562:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.390566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.390568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.390576:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.390582:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.390600:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760779.390728:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 26783744<>26787840 00000100:00100000:3.0:1789760779.390948:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920384:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.443155:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973856 gen 2 00000008:00010000:3.0:1789760779.443244:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760779.443253:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.443265:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.443291:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760779.443321:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920384:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.443541:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6539, 6539] 00000080:00200000:2.0:1789760779.443558:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.443564:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.443579:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26787840 i_blocks 53257 00000020:00200000:2.0:1789760779.443583:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.443593:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.443624:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26787840, count: 4096 00000080:00200000:2.0:1789760779.443636:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26787840, bytes: 4096 00000080:00200000:2.0:1789760779.443646:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.443655:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.443661:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26787840, 26791936) 0 0 00000080:00200000:2.0:1789760779.443665:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.443681:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6540, 6540] 00020000:00200000:2.0:1789760779.443690:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26787840 chunk: [26787840, 26791936) 26791936, 4096 00020000:00200000:2.0:1789760779.443695:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.443715:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6540, 6540] 00000008:00010000:2.0:1789760779.443732:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.443742:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.443754:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26787840, 26791936) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.443768:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26787840=0x198c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.443833:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.443837:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26791936 00000020:00010000:2.0:1789760779.443845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.443849:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.443853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.443856:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.443859:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.443866:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.443873:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.443889:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760779.444012:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 26787840<>26791936 00000100:00100000:3.0:1789760779.444201:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.444218:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920512:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.477423:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973857 gen 2 00000008:00010000:3.0:1789760779.477479:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760779.477488:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.477500:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.477523:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760779.477555:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920512:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.477671:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6540, 6540] 00000080:00200000:2.0:1789760779.477691:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.477698:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.477715:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26791936 i_blocks 53257 00000020:00200000:2.0:1789760779.477719:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.477729:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.477763:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26791936, count: 4096 00000080:00200000:2.0:1789760779.477773:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26791936, bytes: 4096 00000080:00200000:2.0:1789760779.477782:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.477791:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.477794:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26791936, 26796032) 0 0 00000080:00200000:2.0:1789760779.477811:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.477829:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6541, 6541] 00020000:00200000:2.0:1789760779.477838:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26791936 chunk: [26791936, 26796032) 26796032, 4096 00020000:00200000:2.0:1789760779.477843:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.477864:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6541, 6541] 00000008:00010000:2.0:1789760779.477883:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.477889:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.477900:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26791936, 26796032) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.477915:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26791936=0x198d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.477971:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.477974:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26796032 00000020:00010000:2.0:1789760779.477983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.477987:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.477991:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.477995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.477999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.478006:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.478014:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.478295:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:3.0:1789760779.478479:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 26791936<>26796032 00000100:00100000:3.0:1789760779.478538:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920640:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.515809:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973858 gen 2 00000008:00010000:3.0:1789760779.516128:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760779.516137:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.516150:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.516280:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760779.516321:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920640:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.516710:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6541, 6541] 00000080:00200000:2.0:1789760779.516729:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.516736:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.516755:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26796032 i_blocks 53257 00000020:00200000:2.0:1789760779.516759:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.516770:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.516808:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26796032, count: 4096 00000080:00200000:2.0:1789760779.516816:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26796032, bytes: 4096 00000080:00200000:2.0:1789760779.516829:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.516838:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.516840:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26796032, 26800128) 0 0 00000080:00200000:2.0:1789760779.516845:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.517233:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6542, 6542] 00020000:00200000:2.0:1789760779.517243:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26796032 chunk: [26796032, 26800128) 26800128, 4096 00020000:00200000:2.0:1789760779.517248:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.517273:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6542, 6542] 00000008:00010000:2.0:1789760779.517481:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.517489:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.517502:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26796032, 26800128) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.517517:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26796032=0x198e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.517725:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.517729:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26800128 00000020:00010000:2.0:1789760779.517738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.517741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.517744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.517747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.517751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.517760:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.517767:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.517791:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760779.517984:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 26796032<>26800128 00000100:00100000:3.0:1789760779.518073:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495920768:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.558819:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973859 gen 2 00000008:00010000:3.0:1789760779.558869:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760779.558877:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.558899:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.558923:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760779.558952:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495920768:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.558959:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6542, 6542] 00000080:00200000:2.0:1789760779.558980:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.558987:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.559004:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26800128 i_blocks 53257 00000020:00200000:2.0:1789760779.559008:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.559019:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.559059:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26800128, count: 4096 00000080:00200000:2.0:1789760779.559066:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26800128, bytes: 4096 00000080:00200000:2.0:1789760779.559076:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.559086:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.559088:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26800128, 26804224) 0 0 00000080:00200000:2.0:1789760779.559092:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.559109:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6543, 6543] 00020000:00200000:2.0:1789760779.559117:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26800128 chunk: [26800128, 26804224) 26804224, 4096 00020000:00200000:2.0:1789760779.559121:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.559141:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6543, 6543] 00000008:00010000:2.0:1789760779.559159:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.559165:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.559203:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26800128, 26804224) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.559217:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26800128=0x198f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.559271:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.559275:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26804224 00000020:00010000:2.0:1789760779.559283:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.559287:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.559290:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.559293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.559295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.559404:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.559413:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.559430:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760779.559645:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 26800128<>26804224 00000100:00100000:3.0:1789760779.559863:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.609521:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973860 gen 2 00000008:00010000:3.0:1789760779.609572:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760779.609580:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.609592:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.609618:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760779.609648:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495920896:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.610107:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6543, 6543] 00000080:00200000:2.0:1789760779.610126:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.610134:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.610151:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26804224 i_blocks 53257 00000020:00200000:2.0:1789760779.610154:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.610164:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.610229:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26804224, count: 4096 00000080:00200000:2.0:1789760779.610236:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26804224, bytes: 4096 00000080:00200000:2.0:1789760779.610249:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.610257:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.610260:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26804224, 26808320) 0 0 00000080:00200000:2.0:1789760779.610264:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.610281:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6544, 6544] 00020000:00200000:2.0:1789760779.610290:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26804224 chunk: [26804224, 26808320) 26808320, 4096 00020000:00200000:2.0:1789760779.610295:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.610316:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6544, 6544] 00000008:00010000:2.0:1789760779.610342:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.610348:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.610359:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26804224, 26808320) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.610374:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26804224=0x1990000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.610432:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.610435:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26808320 00000020:00010000:2.0:1789760779.610444:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.610448:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.610451:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.610455:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.610458:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.610465:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.610471:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.610490:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760779.611219:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 26804224<>26808320 00000100:00100000:3.0:1789760779.611692:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.611716:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921024:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.661459:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973861 gen 2 00000008:00010000:3.0:1789760779.661512:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760779.661522:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.661533:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.661633:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760779.661662:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921024:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.661775:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6544, 6544] 00000080:00200000:2.0:1789760779.661793:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.661799:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.661815:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26808320 i_blocks 53257 00000020:00200000:2.0:1789760779.661818:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.661828:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.661924:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26808320, count: 4096 00000080:00200000:2.0:1789760779.661932:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26808320, bytes: 4096 00000080:00200000:2.0:1789760779.661942:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.661951:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.661954:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26808320, 26812416) 0 0 00000080:00200000:2.0:1789760779.661958:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.662041:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6545, 6545] 00020000:00200000:2.0:1789760779.662052:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26808320 chunk: [26808320, 26812416) 26812416, 4096 00020000:00200000:2.0:1789760779.662057:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.662137:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6545, 6545] 00000008:00010000:2.0:1789760779.662157:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.662163:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.662206:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26808320, 26812416) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.662222:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26808320=0x1991000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.662281:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.662284:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26812416 00000020:00010000:2.0:1789760779.662293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.662296:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.662299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.662302:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.662305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.662313:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.662320:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.662347:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760779.662468:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 26808320<>26812416 00000100:00100000:3.0:1789760779.662701:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921152:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.693634:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973862 gen 2 00000008:00010000:3.0:1789760779.693746:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760779.693755:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.693766:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.693792:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760779.693823:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921152:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.693944:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6545, 6545] 00000080:00200000:2.0:1789760779.693963:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.693970:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.693986:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26812416 i_blocks 53257 00000020:00200000:2.0:1789760779.693990:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.694001:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.694037:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26812416, count: 4096 00000080:00200000:2.0:1789760779.694045:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26812416, bytes: 4096 00000080:00200000:2.0:1789760779.694054:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.694061:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.694064:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26812416, 26816512) 0 0 00000080:00200000:2.0:1789760779.694068:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.694084:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6546, 6546] 00020000:00200000:2.0:1789760779.694092:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26812416 chunk: [26812416, 26816512) 26816512, 4096 00020000:00200000:2.0:1789760779.694097:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.694115:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6546, 6546] 00000008:00010000:2.0:1789760779.694134:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.694140:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.695132:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26812416, 26816512) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.695155:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26812416=0x1992000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.695240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.695243:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26816512 00000020:00010000:2.0:1789760779.695253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.695257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.695260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.695263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.695266:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.695276:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.695296:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.695320:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760779.695460:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 26812416<>26816512 00000100:00100000:3.0:1789760779.695555:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.695577:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921280:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760779.723412:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760779.732212:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760779.732229:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.732244:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.732267:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760779.732295:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921280:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.732360:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6546, 6546] 00000080:00200000:2.0:1789760779.732378:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.732386:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.732402:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26816512 i_blocks 53257 00000020:00200000:2.0:1789760779.732406:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.732416:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.732450:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26816512, count: 4096 00000080:00200000:2.0:1789760779.732458:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26816512, bytes: 4096 00000080:00200000:2.0:1789760779.732467:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.732476:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.732479:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26816512, 26820608) 0 0 00000080:00200000:2.0:1789760779.732483:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.732509:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6547, 6547] 00020000:00200000:2.0:1789760779.732518:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26816512 chunk: [26816512, 26820608) 26820608, 4096 00020000:00200000:2.0:1789760779.732523:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.732545:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6547, 6547] 00000008:00010000:2.0:1789760779.732565:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.732573:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.732586:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26816512, 26820608) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.732601:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26816512=0x1993000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.732662:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.732664:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26820608 00000020:00010000:2.0:1789760779.732673:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.732677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.732680:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.732684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.732688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.732696:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.732705:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.732722:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760779.732858:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 26816512<>26820608 00000100:00100000:3.0:1789760779.734836:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921408:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760779.768512:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973864 gen 2 00000008:00010000:3.0:1789760779.768585:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760779.768593:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.768603:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.768629:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760779.768659:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921408:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.768997:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6547, 6547] 00000080:00200000:2.0:1789760779.769014:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.769020:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.769037:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26820608 i_blocks 53257 00000020:00200000:2.0:1789760779.769041:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.769051:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.769085:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26820608, count: 4096 00000080:00200000:2.0:1789760779.769091:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26820608, bytes: 4096 00000080:00200000:2.0:1789760779.769101:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.769109:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.769112:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26820608, 26824704) 0 0 00000080:00200000:2.0:1789760779.769116:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.769134:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6548, 6548] 00020000:00200000:2.0:1789760779.769142:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26820608 chunk: [26820608, 26824704) 26824704, 4096 00020000:00200000:2.0:1789760779.769147:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.769196:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6548, 6548] 00000008:00010000:2.0:1789760779.769215:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.769221:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.769234:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26820608, 26824704) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.769248:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26820608=0x1994000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.769305:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.769308:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26824704 00000020:00010000:2.0:1789760779.769317:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.769331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.769335:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.769338:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.769341:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.769349:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.769356:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.769373:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760779.769497:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 26820608<>26824704 00000100:00100000:3.0:1789760779.769701:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.769719:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921536:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760779.812426:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760779.812444:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760779.812457:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760779.812487:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760779.812518:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921536:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760779.812619:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6548, 6548] 00000080:00200000:1.0:1789760779.812636:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.812650:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.812667:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26824704 i_blocks 53257 00000020:00200000:1.0:1789760779.812671:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.812682:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.812718:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26824704, count: 4096 00000080:00200000:1.0:1789760779.812725:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26824704, bytes: 4096 00000080:00200000:1.0:1789760779.812741:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.812750:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.812753:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26824704, 26828800) 0 0 00000080:00200000:1.0:1789760779.812757:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.812774:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6549, 6549] 00020000:00200000:1.0:1789760779.812782:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26824704 chunk: [26824704, 26828800) 26828800, 4096 00020000:00200000:1.0:1789760779.812787:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.812808:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6549, 6549] 00000008:00010000:1.0:1789760779.812827:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.812834:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760779.812847:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26824704, 26828800) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.812862:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26824704=0x1995000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.812915:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.812918:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26828800 00000020:00010000:1.0:1789760779.812927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.812930:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.812934:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.812938:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.812941:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.812948:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.812956:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.812973:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760779.813102:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 26824704<>26828800 00000100:00100000:1.0:1789760779.813150:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760779.813190:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495921664:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760779.853055:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973866 gen 2 00000008:00010000:1.0:1789760779.853131:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760779.853140:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.853151:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.853261:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760779.853289:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495921664:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760779.855112:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6549, 6549] 00000080:00200000:2.0:1789760779.855136:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.855143:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.855162:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26828800 i_blocks 53257 00000020:00200000:2.0:1789760779.855181:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.855194:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.855228:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26828800, count: 4096 00000080:00200000:2.0:1789760779.855235:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26828800, bytes: 4096 00000080:00200000:2.0:1789760779.855246:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.855255:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.855258:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26828800, 26832896) 0 0 00000080:00200000:2.0:1789760779.855262:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.855278:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6550, 6550] 00020000:00200000:2.0:1789760779.855285:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26828800 chunk: [26828800, 26832896) 26832896, 4096 00020000:00200000:2.0:1789760779.855289:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.855310:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6550, 6550] 00000008:00010000:2.0:1789760779.855338:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.855344:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760779.855357:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26828800, 26832896) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.855376:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26828800=0x1996000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.855440:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.855443:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26832896 00000020:00010000:2.0:1789760779.855451:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.855455:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.855458:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.855462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.855466:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.855473:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.855479:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.855506:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760779.855641:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 26828800<>26832896 00000100:00100000:3.0:1789760779.856072:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921792:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760779.891610:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760779.891626:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.891637:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.891776:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760779.891807:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495921792:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760779.891898:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6550, 6550] 00000080:00200000:1.0:1789760779.891915:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.891922:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.891939:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26832896 i_blocks 53257 00000020:00200000:1.0:1789760779.891943:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.891953:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.891988:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26832896, count: 4096 00000080:00200000:1.0:1789760779.891994:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26832896, bytes: 4096 00000080:00200000:1.0:1789760779.892052:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.892061:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.892063:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26832896, 26836992) 0 0 00000080:00200000:1.0:1789760779.892066:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.892079:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6551, 6551] 00020000:00200000:1.0:1789760779.892086:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26832896 chunk: [26832896, 26836992) 26836992, 4096 00020000:00200000:1.0:1789760779.892089:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.892106:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6551, 6551] 00000008:00010000:1.0:1789760779.892122:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.892128:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760779.892141:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26832896, 26836992) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.892156:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26832896=0x1997000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.892238:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.892240:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26836992 00000020:00010000:1.0:1789760779.892248:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.892250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.892252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.892255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.892258:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.892263:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.892269:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.892283:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760779.892624:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 26832896<>26836992 00000100:00100000:0.0:1789760779.893256:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760779.893277:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495921920:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760779.920217:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973868 gen 2 00000008:00010000:1.0:1789760779.920296:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760779.920385:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.920397:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.920418:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760779.920450:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495921920:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760779.920493:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6551, 6551] 00000080:00200000:0.0:1789760779.920510:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.920517:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.920535:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26836992 i_blocks 53257 00000020:00200000:0.0:1789760779.920539:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.920549:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.920583:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26836992, count: 4096 00000080:00200000:0.0:1789760779.920590:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26836992, bytes: 4096 00000080:00200000:0.0:1789760779.920601:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.920610:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.920613:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26836992, 26841088) 0 0 00000080:00200000:0.0:1789760779.920619:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.920637:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6552, 6552] 00020000:00200000:0.0:1789760779.920646:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26836992 chunk: [26836992, 26841088) 26841088, 4096 00020000:00200000:0.0:1789760779.920651:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.920671:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6552, 6552] 00000008:00010000:0.0:1789760779.920691:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.920698:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760779.920709:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26836992, 26841088) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.920722:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26836992=0x1998000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.920779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.920782:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26841088 00000020:00010000:0.0:1789760779.920790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.920794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.920797:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.920801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.920804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.920812:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.920820:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.920836:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760779.921615:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 26836992<>26841088 00000100:00100000:1.0:1789760779.921755:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495922048:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760779.956150:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973869 gen 2 00000008:00010000:1.0:1789760779.956732:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760779.956741:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.956811:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.956828:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000080:00200000:0.0:1789760779.956851:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6552, 6552] 00000100:00100000:1.0:1789760779.956855:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495922048:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760779.956868:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.956874:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.956891:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26841088 i_blocks 53257 00000020:00200000:0.0:1789760779.956895:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.956904:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.956937:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26841088, count: 4096 00000080:00200000:0.0:1789760779.956946:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26841088, bytes: 4096 00000080:00200000:0.0:1789760779.956965:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.957033:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.957036:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26841088, 26845184) 0 0 00000080:00200000:0.0:1789760779.957041:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.957059:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6553, 6553] 00020000:00200000:0.0:1789760779.957066:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26841088 chunk: [26841088, 26845184) 26845184, 4096 00020000:00200000:0.0:1789760779.957072:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.957092:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6553, 6553] 00000008:00010000:0.0:1789760779.957112:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.957118:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760779.957130:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26841088, 26845184) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.957144:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26841088=0x1999000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.957222:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.957225:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26845184 00000020:00010000:0.0:1789760779.957233:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.957237:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.957239:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.957242:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.957245:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.957251:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.957257:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.957268:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760779.957401:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 26841088<>26845184 00000100:00100000:1.0:1789760779.957512:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495922176:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760779.991660:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760779.991677:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.991690:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.991713:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760779.991743:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495922176:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760779.991960:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6553, 6553] 00000080:00200000:0.0:1789760779.991979:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.991986:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.992001:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26845184 i_blocks 53257 00000020:00200000:0.0:1789760779.992005:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.992015:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.992049:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26845184, count: 4096 00000080:00200000:0.0:1789760779.992055:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26845184, bytes: 4096 00000080:00200000:0.0:1789760779.992065:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.992073:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.992075:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26845184, 26849280) 0 0 00000080:00200000:0.0:1789760779.992079:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.992095:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6554, 6554] 00020000:00200000:0.0:1789760779.992104:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26845184 chunk: [26845184, 26849280) 26849280, 4096 00020000:00200000:0.0:1789760779.992108:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.992130:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6554, 6554] 00000008:00010000:0.0:1789760779.992149:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.992155:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760779.992166:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26845184, 26849280) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.992215:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26845184=0x199a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.992268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.992271:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26849280 00000020:00010000:0.0:1789760779.992280:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.992284:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.992286:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.992290:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.992293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.992300:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.992308:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.992338:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760779.992465:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 26845184<>26849280 00000100:00100000:1.0:1789760779.992618:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760779.992636:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495922304:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760780.018948:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973871 gen 2 00000008:00010000:1.0:1789760780.019015:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760780.019023:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.019033:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.019057:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760780.019090:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495922304:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760780.019260:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6554, 6554] 00000080:00200000:0.0:1789760780.019278:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760780.019285:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760780.019370:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26849280 i_blocks 53257 00000020:00200000:0.0:1789760780.019376:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760780.019386:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760780.019421:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26849280, count: 4096 00000080:00200000:0.0:1789760780.019429:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26849280, bytes: 4096 00000080:00200000:0.0:1789760780.019440:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760780.019449:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760780.019451:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26849280, 26853376) 0 0 00000080:00200000:0.0:1789760780.019455:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760780.019472:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6555, 6555] 00020000:00200000:0.0:1789760780.019480:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26849280 chunk: [26849280, 26853376) 26853376, 4096 00020000:00200000:0.0:1789760780.019485:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760780.019506:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6555, 6555] 00000008:00010000:0.0:1789760780.019525:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760780.019531:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760780.019542:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26849280, 26853376) append 0 append lockpos 0 00000080:00200000:0.0:1789760780.019559:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26849280=0x199b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760780.019623:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760780.019626:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26853376 00000020:00010000:0.0:1789760780.019634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760780.019638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.019708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.019712:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760780.019716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760780.019724:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760780.019730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760780.019747:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760780.019949:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 26849280<>26853376 00000100:00100000:1.0:1789760780.020036:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495922432:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760780.053571:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973872 gen 2 00000008:00010000:1.0:1789760780.053624:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760780.053633:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.053706:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.053790:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760780.053821:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495922432:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.054082:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6555, 6555] 00000080:00200000:2.0:1789760780.054099:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.054105:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.054121:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26853376 i_blocks 53257 00000020:00200000:2.0:1789760780.054125:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.054135:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.054204:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26853376, count: 4096 00000080:00200000:2.0:1789760780.054212:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26853376, bytes: 4096 00000080:00200000:2.0:1789760780.054223:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.054232:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.054235:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26853376, 26857472) 0 0 00000080:00200000:2.0:1789760780.054240:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.054258:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6556, 6556] 00020000:00200000:2.0:1789760780.054266:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26853376 chunk: [26853376, 26857472) 26857472, 4096 00020000:00200000:2.0:1789760780.054397:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.054418:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6556, 6556] 00000008:00010000:2.0:1789760780.054436:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.054441:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.054453:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26853376, 26857472) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.054467:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26853376=0x199c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.054525:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.054528:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26857472 00000020:00010000:2.0:1789760780.054537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.054541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.054544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.054547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.054550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.054558:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.054565:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.054583:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760780.054839:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 26853376<>26857472 00000100:00100000:3.0:1789760780.055000:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495922560:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760780.089024:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:2.0:1789760780.089040:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760780.089053:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760780.089140:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:2.0:1789760780.089230:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495922560:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760780.090951:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6556, 6556] 00000080:00200000:1.0:1789760780.090981:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.090989:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.091009:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26857472 i_blocks 53257 00000020:00200000:1.0:1789760780.091015:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.091029:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.091066:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26857472, count: 4096 00000080:00200000:1.0:1789760780.091074:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26857472, bytes: 4096 00000080:00200000:1.0:1789760780.091089:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.091099:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.091101:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26857472, 26861568) 0 0 00000080:00200000:1.0:1789760780.091258:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.091278:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6557, 6557] 00020000:00200000:1.0:1789760780.091287:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26857472 chunk: [26857472, 26861568) 26861568, 4096 00020000:00200000:1.0:1789760780.091292:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.091400:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6557, 6557] 00000008:00010000:1.0:1789760780.091423:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.091429:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760780.091442:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26857472, 26861568) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.091456:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26857472=0x199d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.091528:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.091531:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26861568 00000020:00010000:1.0:1789760780.091541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.091544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.091547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.091551:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.091554:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.091563:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.091571:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.091601:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760780.091742:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 26857472<>26861568 00000100:00100000:0.0:1789760780.091855:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495922688:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760780.121623:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973873 gen 2 00000100:00100000:1.0:1789760780.121667:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504c380 x1876699495922688/t4294973874(4294973874) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760796 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760780.121719:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760780.121727:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.121738:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.122077:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760780.122106:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495922688:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.122645:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6557, 6557] 00000080:00200000:2.0:1789760780.122666:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.122673:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.122691:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26861568 i_blocks 53257 00000020:00200000:2.0:1789760780.122697:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.122709:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.122752:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26861568, count: 4096 00000080:00200000:2.0:1789760780.122759:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26861568, bytes: 4096 00000080:00200000:2.0:1789760780.122773:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.122782:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.122785:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26861568, 26865664) 0 0 00000080:00200000:2.0:1789760780.122790:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.122808:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6558, 6558] 00020000:00200000:2.0:1789760780.122817:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26861568 chunk: [26861568, 26865664) 26865664, 4096 00020000:00200000:2.0:1789760780.122822:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.122845:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6558, 6558] 00000008:00010000:2.0:1789760780.122866:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.122872:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.123214:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26861568, 26865664) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.123233:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26861568=0x199e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.123305:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.123309:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26865664 00000020:00010000:2.0:1789760780.123330:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.123334:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.123337:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.123340:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.123472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.123483:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.123492:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.123517:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760780.123693:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 26861568<>26865664 00000100:00100000:3.0:1789760780.123794:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495922816:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.170988:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973875 gen 2 00000008:00010000:3.0:1789760780.171051:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760780.171645:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.171659:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.171757:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760780.171792:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495922816:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.171881:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6558, 6558] 00000080:00200000:2.0:1789760780.171898:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.171905:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.171921:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26865664 i_blocks 53257 00000020:00200000:2.0:1789760780.171924:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.171935:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.171970:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26865664, count: 4096 00000080:00200000:2.0:1789760780.171977:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26865664, bytes: 4096 00000080:00200000:2.0:1789760780.171989:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.171998:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.172000:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26865664, 26869760) 0 0 00000080:00200000:2.0:1789760780.172005:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.172022:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6559, 6559] 00020000:00200000:2.0:1789760780.172030:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26865664 chunk: [26865664, 26869760) 26869760, 4096 00020000:00200000:2.0:1789760780.172035:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.172056:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6559, 6559] 00000008:00010000:2.0:1789760780.172280:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.172286:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.172299:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26865664, 26869760) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.172313:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26865664=0x199f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.172379:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.172382:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26869760 00000020:00010000:2.0:1789760780.172391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.172394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.172510:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.172517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.172521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.172530:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.172537:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.172549:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760780.172872:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 26865664<>26869760 00000100:00100000:3.0:1789760780.172954:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495922944:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.236566:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973876 gen 2 00000008:00010000:3.0:1789760780.236623:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760780.236632:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.236643:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.236747:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760780.236776:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495922944:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760780.236996:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6559, 6559] 00000080:00200000:1.0:1789760780.237013:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.237019:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.237085:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26869760 i_blocks 53257 00000020:00200000:1.0:1789760780.237089:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.237098:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.237134:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26869760, count: 4096 00000080:00200000:1.0:1789760780.237142:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26869760, bytes: 4096 00000080:00200000:1.0:1789760780.237157:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.237191:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.237194:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26869760, 26873856) 0 0 00000080:00200000:1.0:1789760780.237199:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.237215:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6560, 6560] 00020000:00200000:1.0:1789760780.237224:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26869760 chunk: [26869760, 26873856) 26873856, 4096 00020000:00200000:1.0:1789760780.237228:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.237250:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6560, 6560] 00000008:00010000:1.0:1789760780.237267:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.237272:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760780.237282:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26869760, 26873856) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.237296:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26869760=0x19a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.237391:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.237394:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26873856 00000020:00010000:1.0:1789760780.237469:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.237474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.237478:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.237482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.237485:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.237493:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.237499:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.237517:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760780.237635:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 26869760<>26873856 00000100:00100000:0.0:1789760780.237718:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495923072:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760780.270710:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973877 gen 2 00000008:00010000:1.0:1789760780.270758:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760780.270768:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.270778:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.270868:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760780.270897:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495923072:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.271234:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6560, 6560] 00000080:00200000:2.0:1789760780.271252:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.271257:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.271274:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26873856 i_blocks 53257 00000020:00200000:2.0:1789760780.271278:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.271288:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.271387:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26873856, count: 4096 00000080:00200000:2.0:1789760780.271395:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26873856, bytes: 4096 00000080:00200000:2.0:1789760780.271406:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.271415:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.271418:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26873856, 26877952) 0 0 00000080:00200000:2.0:1789760780.271423:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.271439:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6561, 6561] 00020000:00200000:2.0:1789760780.271447:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26873856 chunk: [26873856, 26877952) 26877952, 4096 00020000:00200000:2.0:1789760780.271452:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.271471:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6561, 6561] 00000008:00010000:2.0:1789760780.271487:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.271493:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.271504:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26873856, 26877952) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.271518:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26873856=0x19a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.271568:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.271571:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26877952 00000020:00010000:2.0:1789760780.271579:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.271583:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.271586:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.271590:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.271593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.271600:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.271607:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.271623:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760780.271803:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 26873856<>26877952 00000100:00100000:3.0:1789760780.271884:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923200:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760780.304226:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760780.304241:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.304252:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.304278:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000080:00200000:2.0:1789760780.304355:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6561, 6561] 00000100:00100000:3.0:1789760780.304373:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923200:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.304375:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.304384:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.304400:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26877952 i_blocks 53257 00000020:00200000:2.0:1789760780.304404:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.304415:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.304460:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26877952, count: 4096 00000080:00200000:2.0:1789760780.304467:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26877952, bytes: 4096 00000080:00200000:2.0:1789760780.304477:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.304486:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.304488:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26877952, 26882048) 0 0 00000080:00200000:2.0:1789760780.304493:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.304510:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6562, 6562] 00020000:00200000:2.0:1789760780.304518:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26877952 chunk: [26877952, 26882048) 26882048, 4096 00020000:00200000:2.0:1789760780.304523:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.304543:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6562, 6562] 00000008:00010000:2.0:1789760780.304560:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.304566:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.304578:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26877952, 26882048) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.304592:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26877952=0x19a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.304643:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.304646:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26882048 00000020:00010000:2.0:1789760780.304655:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.304659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.304662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.304666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.304669:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.304676:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.304683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.304699:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760780.304812:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26877952<>26882048 00000100:00100000:3.0:1789760780.305082:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760780.305101:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923328:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.349628:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973878 gen 2 00000100:00100000:3.0:1789760780.349673:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205265500 x1876699495923328/t4294973879(4294973879) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760796 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760780.349722:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760780.349730:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.349740:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.349765:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760780.349793:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923328:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.350007:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6562, 6562] 00000080:00200000:2.0:1789760780.350023:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.350030:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.350045:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26882048 i_blocks 53257 00000020:00200000:2.0:1789760780.350048:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.350058:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.350092:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26882048, count: 4096 00000080:00200000:2.0:1789760780.350099:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26882048, bytes: 4096 00000080:00200000:2.0:1789760780.350110:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.350119:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.350122:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26882048, 26886144) 0 0 00000080:00200000:2.0:1789760780.350127:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.350141:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6563, 6563] 00020000:00200000:2.0:1789760780.350150:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26882048 chunk: [26882048, 26886144) 26886144, 4096 00020000:00200000:2.0:1789760780.350155:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.350199:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6563, 6563] 00000008:00010000:2.0:1789760780.350219:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.350225:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.350237:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26882048, 26886144) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.350252:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26882048=0x19a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.350309:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.350312:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26886144 00000020:00010000:2.0:1789760780.350331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.350335:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.350338:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.350342:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.350345:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.350352:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.350359:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.350376:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760780.350970:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26882048<>26886144 00000100:00100000:3.0:1789760780.351103:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923456:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.386833:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973880 gen 2 00000008:00010000:3.0:1789760780.386893:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760780.386902:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.386915:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.386939:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760780.386972:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923456:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.387046:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6563, 6563] 00000080:00200000:2.0:1789760780.387064:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.387072:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.387087:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26886144 i_blocks 53257 00000020:00200000:2.0:1789760780.387092:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.387102:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.387141:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26886144, count: 4096 00000080:00200000:2.0:1789760780.387150:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26886144, bytes: 4096 00000080:00200000:2.0:1789760780.387160:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.387202:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.387205:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26886144, 26890240) 0 0 00000080:00200000:2.0:1789760780.387210:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.387231:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6564, 6564] 00020000:00200000:2.0:1789760780.387241:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26886144 chunk: [26886144, 26890240) 26890240, 4096 00020000:00200000:2.0:1789760780.387247:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.387270:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6564, 6564] 00000008:00010000:2.0:1789760780.387289:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.387295:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.387325:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26886144, 26890240) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.387340:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26886144=0x19a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.387402:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.387405:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26890240 00000020:00010000:2.0:1789760780.387414:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.387417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.387420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.387423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.387426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.387434:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.387441:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.387457:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760780.387593:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 26886144<>26890240 00000100:00100000:3.0:1789760780.387762:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760780.387781:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923584:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760780.421557:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760780.421574:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.421587:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.421611:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760780.421651:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923584:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.422146:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6564, 6564] 00000080:00200000:2.0:1789760780.422163:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.422207:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.422226:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26890240 i_blocks 53257 00000020:00200000:2.0:1789760780.422231:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.422242:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.422278:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26890240, count: 4096 00000080:00200000:2.0:1789760780.422286:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26890240, bytes: 4096 00000080:00200000:2.0:1789760780.422298:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.422308:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.422311:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26890240, 26894336) 0 0 00000080:00200000:2.0:1789760780.422327:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.422348:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6565, 6565] 00020000:00200000:2.0:1789760780.422356:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26890240 chunk: [26890240, 26894336) 26894336, 4096 00020000:00200000:2.0:1789760780.422361:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.422381:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6565, 6565] 00000008:00010000:2.0:1789760780.422398:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.422405:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.422417:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26890240, 26894336) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.422431:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26890240=0x19a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.422491:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.422494:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26894336 00000020:00010000:2.0:1789760780.422503:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.422507:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.422510:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.422514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.422517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.422526:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.422534:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.422552:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760780.424444:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 26890240<>26894336 00000100:00100000:3.0:1789760780.424756:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923712:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.467882:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973882 gen 2 00000008:00010000:3.0:1789760780.468008:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760780.468017:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.468028:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.468052:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760780.468082:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923712:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.468196:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6565, 6565] 00000080:00200000:2.0:1789760780.468214:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.468220:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.468237:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26894336 i_blocks 53257 00000020:00200000:2.0:1789760780.468240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.468250:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.468283:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26894336, count: 4096 00000080:00200000:2.0:1789760780.468290:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26894336, bytes: 4096 00000080:00200000:2.0:1789760780.468299:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.468308:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.468311:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26894336, 26898432) 0 0 00000080:00200000:2.0:1789760780.468328:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.468344:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6566, 6566] 00020000:00200000:2.0:1789760780.468977:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26894336 chunk: [26894336, 26898432) 26898432, 4096 00020000:00200000:2.0:1789760780.468984:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.469007:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6566, 6566] 00000008:00010000:2.0:1789760780.469029:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.469035:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.469047:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26894336, 26898432) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.469061:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26894336=0x19a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.469123:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.469126:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26898432 00000020:00010000:2.0:1789760780.469134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.469138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.469141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.469144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.469147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.469154:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.469161:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.469762:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760780.469909:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 26894336<>26898432 00000100:00100000:2.0:1789760780.470834:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495923840:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760780.515022:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973883 gen 2 00000008:00010000:2.0:1789760780.515073:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760780.515081:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760780.515093:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760780.515160:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760780.515226:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495923840:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760780.515313:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6566, 6566] 00000080:00200000:3.0:1789760780.515410:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.515418:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.515435:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26898432 i_blocks 53257 00000020:00200000:3.0:1789760780.515439:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.515448:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.515564:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26898432, count: 4096 00000080:00200000:3.0:1789760780.515572:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26898432, bytes: 4096 00000080:00200000:3.0:1789760780.515583:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.515591:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760780.515594:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26898432, 26902528) 0 0 00000080:00200000:3.0:1789760780.515599:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.515617:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6567, 6567] 00020000:00200000:3.0:1789760780.515625:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26898432 chunk: [26898432, 26902528) 26902528, 4096 00020000:00200000:3.0:1789760780.515630:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.515651:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6567, 6567] 00000008:00010000:3.0:1789760780.515669:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.515675:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760780.515687:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26898432, 26902528) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.515702:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26898432=0x19a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.515765:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.515768:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26902528 00000020:00010000:3.0:1789760780.515776:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.515780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.515784:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.515787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.515791:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.515798:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.515805:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.515822:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760780.515957:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 26898432<>26902528 00000100:00100000:2.0:1789760780.516036:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923968:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760780.547569:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973884 gen 2 00000008:00010000:2.0:1789760780.547625:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:2.0:1789760780.547633:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760780.547653:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760780.547718:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:2.0:1789760780.547748:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495923968:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760780.548441:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6567, 6567] 00000080:00200000:3.0:1789760780.548459:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.548465:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.548481:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26902528 i_blocks 53257 00000020:00200000:3.0:1789760780.548486:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.548495:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.548529:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26902528, count: 4096 00000080:00200000:3.0:1789760780.548537:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26902528, bytes: 4096 00000080:00200000:3.0:1789760780.548547:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.548555:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760780.548557:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26902528, 26906624) 0 0 00000080:00200000:3.0:1789760780.548561:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.548578:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6568, 6568] 00020000:00200000:3.0:1789760780.548587:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26902528 chunk: [26902528, 26906624) 26906624, 4096 00020000:00200000:3.0:1789760780.548592:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.548613:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6568, 6568] 00000008:00010000:3.0:1789760780.548630:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.548714:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760780.548729:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26902528, 26906624) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.548744:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26902528=0x19a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.548800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.548803:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26906624 00000020:00010000:3.0:1789760780.548812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.548815:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.548818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.548822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.548824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.548832:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.548838:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.549831:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760780.549980:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 26902528<>26906624 00000100:00100000:2.0:1789760780.550086:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495924096:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760780.579036:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:2.0:1789760780.579051:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760780.579064:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760780.579574:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:2.0:1789760780.579605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495924096:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760780.580119:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6568, 6568] 00000080:00200000:3.0:1789760780.580137:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.580144:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.580297:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26906624 i_blocks 53257 00000020:00200000:2.0:1789760780.580303:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.580313:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.580361:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26906624, count: 4096 00000080:00200000:2.0:1789760780.580369:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26906624, bytes: 4096 00000080:00200000:2.0:1789760780.580381:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.580391:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.580393:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26906624, 26910720) 0 0 00000080:00200000:2.0:1789760780.580398:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.580416:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6569, 6569] 00020000:00200000:2.0:1789760780.580424:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26906624 chunk: [26906624, 26910720) 26910720, 4096 00020000:00200000:2.0:1789760780.580429:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.580450:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6569, 6569] 00000008:00010000:2.0:1789760780.580469:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.580474:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.580486:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26906624, 26910720) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.580498:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26906624=0x19a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.580545:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.580547:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26910720 00000020:00010000:2.0:1789760780.580553:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.580556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.580559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.580561:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.580563:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.580568:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.580573:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.580611:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760780.580716:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 26906624<>26910720 00000100:00100000:3.0:1789760780.580779:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495924224:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.605525:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973886 gen 2 00000008:00010000:3.0:1789760780.605751:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760780.605760:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.605771:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.605860:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760780.605903:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495924224:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760780.607044:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6569, 6569] 00000080:00200000:1.0:1789760780.607069:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.607076:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.607094:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26910720 i_blocks 53257 00000020:00200000:1.0:1789760780.607100:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.607113:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.607154:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26910720, count: 4096 00000080:00200000:1.0:1789760780.607162:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26910720, bytes: 4096 00000080:00200000:1.0:1789760780.607190:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.607200:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.607203:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26910720, 26914816) 0 0 00000080:00200000:1.0:1789760780.607208:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.607228:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6570, 6570] 00020000:00200000:1.0:1789760780.607237:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26910720 chunk: [26910720, 26914816) 26914816, 4096 00020000:00200000:1.0:1789760780.607242:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.607267:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6570, 6570] 00000008:00010000:1.0:1789760780.607288:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.607294:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760780.607307:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26910720, 26914816) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.607540:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26910720=0x19aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.607692:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.607696:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26914816 00000020:00010000:1.0:1789760780.607705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.607708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.607712:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.607716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.607719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.607728:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.607735:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.607765:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760780.608038:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 26910720<>26914816 00000100:00100000:1.0:1789760780.608104:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760780.608129:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495924352:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760780.645605:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973887 gen 2 00000008:00010000:1.0:1789760780.645657:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760780.645666:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.645678:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.645765:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760780.645798:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495924352:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760780.646143:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6570, 6570] 00000080:00200000:3.0:1789760780.646161:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.646243:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.646410:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26914816 i_blocks 53257 00000020:00200000:1.0:1789760780.646416:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.646426:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.646463:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26914816, count: 4096 00000080:00200000:1.0:1789760780.646471:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26914816, bytes: 4096 00000080:00200000:1.0:1789760780.646605:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.646616:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.646618:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26914816, 26918912) 0 0 00000080:00200000:1.0:1789760780.646624:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.646642:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6571, 6571] 00020000:00200000:1.0:1789760780.646650:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26914816 chunk: [26914816, 26918912) 26918912, 4096 00020000:00200000:1.0:1789760780.646655:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.646678:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6571, 6571] 00000008:00010000:1.0:1789760780.646696:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.646702:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760780.646777:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26914816, 26918912) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.646791:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26914816=0x19ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.647019:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.647022:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26918912 00000020:00010000:1.0:1789760780.647032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.647035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.647039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.647042:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.647045:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.647052:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.647059:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.647079:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760780.647236:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 26914816<>26918912 00000100:00100000:0.0:1789760780.647331:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495924480:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760780.688520:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973888 gen 2 00000008:00010000:1.0:1789760780.688570:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760780.688579:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.688591:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.688617:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760780.688650:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495924480:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.688668:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6571, 6571] 00000080:00200000:2.0:1789760780.688687:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.688695:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.688711:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26918912 i_blocks 53257 00000020:00200000:2.0:1789760780.688716:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.688726:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.688761:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26918912, count: 4096 00000080:00200000:2.0:1789760780.688769:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26918912, bytes: 4096 00000080:00200000:2.0:1789760780.688778:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.688788:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.688790:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26918912, 26923008) 0 0 00000080:00200000:2.0:1789760780.688794:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.688810:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6572, 6572] 00020000:00200000:2.0:1789760780.688819:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26918912 chunk: [26918912, 26923008) 26923008, 4096 00020000:00200000:2.0:1789760780.688825:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.688847:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6572, 6572] 00000008:00010000:2.0:1789760780.688864:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.688870:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.689279:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26918912, 26923008) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.689370:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26918912=0x19ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.689439:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.689442:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26923008 00000020:00010000:2.0:1789760780.689452:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.689456:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.689460:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.689464:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.689466:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.689475:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.689483:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.689580:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760780.689722:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 26918912<>26923008 00000100:00100000:3.0:1789760780.690060:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495924608:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760780.721776:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760780.721794:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.721810:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.721847:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000080:00200000:2.0:1789760780.722153:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6572, 6572] 00000080:00200000:2.0:1789760780.722211:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.722217:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.722234:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26923008 i_blocks 53257 00000020:00200000:2.0:1789760780.722238:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.722248:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.722285:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26923008, count: 4096 00000100:00100000:3.0:1789760780.722297:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495924608:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.722396:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26923008, bytes: 4096 00000080:00200000:2.0:1789760780.722409:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.722418:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.722420:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26923008, 26927104) 0 0 00000080:00200000:2.0:1789760780.722425:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.722442:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6573, 6573] 00020000:00200000:2.0:1789760780.722451:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26923008 chunk: [26923008, 26927104) 26927104, 4096 00020000:00200000:2.0:1789760780.722456:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.722477:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6573, 6573] 00000008:00010000:2.0:1789760780.722497:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.722503:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.722515:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26923008, 26927104) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.722530:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26923008=0x19ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.722591:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.722594:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26927104 00000020:00010000:2.0:1789760780.722603:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.722607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.722610:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.722614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.722617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.722625:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.722632:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.722653:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760780.722785:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26923008<>26927104 00000100:00100000:3.0:1789760780.723053:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495924736:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760780.749496:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760780.766446:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973890 gen 2 00000008:00010000:2.0:1789760780.766513:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:2.0:1789760780.766522:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760780.766533:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760780.766568:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:2.0:1789760780.766599:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495924736:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760780.766628:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6573, 6573] 00000080:00200000:1.0:1789760780.766648:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.766654:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.766672:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26927104 i_blocks 53257 00000020:00200000:1.0:1789760780.766676:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.766687:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.766723:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26927104, count: 4096 00000080:00200000:1.0:1789760780.766731:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26927104, bytes: 4096 00000080:00200000:1.0:1789760780.766743:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.766753:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.766756:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26927104, 26931200) 0 0 00000080:00200000:1.0:1789760780.766761:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.766777:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6574, 6574] 00020000:00200000:1.0:1789760780.766786:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26927104 chunk: [26927104, 26931200) 26931200, 4096 00020000:00200000:1.0:1789760780.766791:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.766811:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6574, 6574] 00000008:00010000:1.0:1789760780.766841:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.766848:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760780.766860:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26927104, 26931200) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.766877:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26927104=0x19ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.766943:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.766946:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26931200 00000020:00010000:1.0:1789760780.766957:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.766960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.766963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.766967:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.766970:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.766977:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.766985:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.767003:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760780.767148:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 26927104<>26931200 00000100:00100000:1.0:1789760780.767227:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760780.767243:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495924864:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760780.802807:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973891 gen 2 00000008:00010000:1.0:1789760780.803013:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760780.803321:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.803333:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.803413:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760780.803443:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495924864:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.803527:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6574, 6574] 00000080:00200000:2.0:1789760780.803544:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.803551:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.803568:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26931200 i_blocks 53257 00000020:00200000:2.0:1789760780.803572:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.803583:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.803620:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26931200, count: 4096 00000080:00200000:2.0:1789760780.803627:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26931200, bytes: 4096 00000080:00200000:2.0:1789760780.812256:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.812281:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.812284:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26931200, 26935296) 0 0 00000080:00200000:2.0:1789760780.812294:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.812334:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6575, 6575] 00020000:00200000:2.0:1789760780.812346:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26931200 chunk: [26931200, 26935296) 26935296, 4096 00020000:00200000:2.0:1789760780.812352:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.812382:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6575, 6575] 00000008:00010000:2.0:1789760780.812405:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.812412:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.812426:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26931200, 26935296) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.812441:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26931200=0x19af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.812521:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.812525:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26935296 00000020:00010000:2.0:1789760780.812535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.812539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.812542:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.812546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.812549:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.812558:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.812567:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.812599:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760780.812812:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 26931200<>26935296 00000100:00100000:3.0:1789760780.813050:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495924992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.866882:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973892 gen 2 00000008:00010000:3.0:1789760780.866935:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760780.866944:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.866955:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.866982:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760780.867011:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495924992:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.867840:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6575, 6575] 00000080:00200000:2.0:1789760780.867860:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.867867:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.867883:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26935296 i_blocks 53257 00000020:00200000:2.0:1789760780.867888:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.867898:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.867932:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26935296, count: 4096 00000080:00200000:2.0:1789760780.867939:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26935296, bytes: 4096 00000080:00200000:2.0:1789760780.867952:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.867961:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.867964:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26935296, 26939392) 0 0 00000080:00200000:2.0:1789760780.867968:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.867985:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6576, 6576] 00020000:00200000:2.0:1789760780.867993:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26935296 chunk: [26935296, 26939392) 26939392, 4096 00020000:00200000:2.0:1789760780.867998:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.868020:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6576, 6576] 00000008:00010000:2.0:1789760780.868040:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.868047:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.868059:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26935296, 26939392) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.868074:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26935296=0x19b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.868140:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.868143:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26939392 00000020:00010000:2.0:1789760780.868152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.868155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.868158:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.868162:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.868165:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.868223:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.868241:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.868258:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760780.868394:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 26935296<>26939392 00000100:00100000:3.0:1789760780.869618:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495925120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.904903:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973893 gen 2 00000008:00010000:3.0:1789760780.904950:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760780.904958:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.904968:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.905078:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760780.905107:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495925120:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760780.905226:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6576, 6576] 00000080:00200000:2.0:1789760780.905303:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.905310:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.905335:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26939392 i_blocks 53257 00000020:00200000:2.0:1789760780.905339:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.905349:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.905381:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26939392, count: 4096 00000080:00200000:2.0:1789760780.905388:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26939392, bytes: 4096 00000080:00200000:2.0:1789760780.905398:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.905408:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.905411:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26939392, 26943488) 0 0 00000080:00200000:2.0:1789760780.905416:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.905431:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6577, 6577] 00020000:00200000:2.0:1789760780.905441:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26939392 chunk: [26939392, 26943488) 26943488, 4096 00020000:00200000:2.0:1789760780.905446:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.905466:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6577, 6577] 00000008:00010000:2.0:1789760780.905485:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.905491:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760780.905503:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26939392, 26943488) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.905517:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26939392=0x19b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.905746:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.905750:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26943488 00000020:00010000:2.0:1789760780.905760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.905763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.905766:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.905769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.905772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.905781:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.905788:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.905807:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760780.905928:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 26939392<>26943488 00000100:00100000:3.0:1789760780.906001:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495925248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760780.970076:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973894 gen 2 00000008:00010000:3.0:1789760780.970129:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760780.970139:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.970151:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.970209:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760780.970238:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495925248:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760780.970411:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6577, 6577] 00000080:00200000:0.0:1789760780.970429:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760780.970437:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760780.970464:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26943488 i_blocks 53257 00000020:00200000:0.0:1789760780.970468:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760780.970478:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760780.970514:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26943488, count: 4096 00000080:00200000:0.0:1789760780.970522:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26943488, bytes: 4096 00000080:00200000:0.0:1789760780.970532:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760780.970541:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760780.970543:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26943488, 26947584) 0 0 00000080:00200000:0.0:1789760780.970548:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760780.970565:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6578, 6578] 00020000:00200000:0.0:1789760780.970573:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26943488 chunk: [26943488, 26947584) 26947584, 4096 00020000:00200000:0.0:1789760780.970579:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760780.970598:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6578, 6578] 00000008:00010000:0.0:1789760780.970616:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760780.970622:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760780.970646:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26943488, 26947584) append 0 append lockpos 0 00000080:00200000:0.0:1789760780.970662:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26943488=0x19b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760780.970730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760780.970733:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26947584 00000020:00010000:0.0:1789760780.970742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760780.970745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.970749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.970753:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760780.970756:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760780.970763:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760780.970771:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760780.970788:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760780.970910:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 26943488<>26947584 00000100:00100000:0.0:1789760780.970960:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760780.970976:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495925376:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.001875:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973895 gen 2 00000008:00010000:1.0:1789760781.001926:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760781.001935:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.001945:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.001965:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760781.001996:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495925376:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.002039:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6578, 6578] 00000080:00200000:0.0:1789760781.002055:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.002062:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.002078:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26947584 i_blocks 53257 00000020:00200000:0.0:1789760781.002082:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.002092:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.002125:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26947584, count: 4096 00000080:00200000:0.0:1789760781.002133:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26947584, bytes: 4096 00000080:00200000:0.0:1789760781.002143:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.002153:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.002155:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26947584, 26951680) 0 0 00000080:00200000:0.0:1789760781.002160:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.002211:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6579, 6579] 00020000:00200000:0.0:1789760781.002220:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26947584 chunk: [26947584, 26951680) 26951680, 4096 00020000:00200000:0.0:1789760781.002226:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.002246:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6579, 6579] 00000008:00010000:0.0:1789760781.002265:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.002272:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.002283:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26947584, 26951680) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.002308:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26947584=0x19b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.002490:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.002493:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26951680 00000020:00010000:0.0:1789760781.002504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.002507:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.002510:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.002513:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.002516:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.002524:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.002531:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.002549:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760781.002694:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 26947584<>26951680 00000100:00100000:1.0:1789760781.002882:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495925504:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.043509:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973896 gen 2 00000008:00010000:1.0:1789760781.043559:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760781.043568:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.043581:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.043599:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760781.043629:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495925504:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.044002:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6579, 6579] 00000080:00200000:0.0:1789760781.044019:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.044025:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.044043:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26951680 i_blocks 53257 00000020:00200000:0.0:1789760781.044047:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.044056:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.044091:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26951680, count: 4096 00000080:00200000:0.0:1789760781.044098:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26951680, bytes: 4096 00000080:00200000:0.0:1789760781.044108:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.044117:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.044120:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26951680, 26955776) 0 0 00000080:00200000:0.0:1789760781.044125:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.044140:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6580, 6580] 00020000:00200000:0.0:1789760781.044148:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26951680 chunk: [26951680, 26955776) 26955776, 4096 00020000:00200000:0.0:1789760781.044154:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.044196:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6580, 6580] 00000008:00010000:0.0:1789760781.044214:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.044220:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.044232:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26951680, 26955776) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.044258:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26951680=0x19b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.045608:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.045618:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26955776 00000020:00010000:0.0:1789760781.045697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.045703:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.045707:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.045711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.045714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.045728:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.045737:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.045763:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760781.046064:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 26951680<>26955776 00000100:00100000:1.0:1789760781.046206:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760781.046233:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495925632:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760781.085091:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760781.085108:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.085120:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.085142:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000080:00200000:0.0:1789760781.085274:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6580, 6580] 00000080:00200000:0.0:1789760781.085305:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.085321:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.085337:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26955776 i_blocks 53257 00000020:00200000:0.0:1789760781.085341:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.085350:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.085456:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26955776, count: 4096 00000080:00200000:0.0:1789760781.085466:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26955776, bytes: 4096 00000080:00200000:0.0:1789760781.085477:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.085487:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.085489:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26955776, 26959872) 0 0 00000080:00200000:0.0:1789760781.085494:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.085511:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6581, 6581] 00020000:00200000:0.0:1789760781.085519:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26955776 chunk: [26955776, 26959872) 26959872, 4096 00020000:00200000:0.0:1789760781.085524:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.085543:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6581, 6581] 00000008:00010000:0.0:1789760781.085562:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.085569:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.085582:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26955776, 26959872) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.085597:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26955776=0x19b5000, pages 1 (max 131072), lockless, parallel 00000100:00100000:1.0:1789760781.085764:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495925632:192.168.203.109@tcp:4:dd.0 00000020:00200000:0.0:1789760781.086071:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.086074:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26959872 00000020:00010000:0.0:1789760781.086084:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.086088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.086091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.086095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.086098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.086106:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.086114:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.086130:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760781.086276:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 26955776<>26959872 00000100:00100000:1.0:1789760781.086675:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495925760:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.120426:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973898 gen 2 00000008:00010000:1.0:1789760781.120491:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760781.120500:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.120511:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.120533:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760781.120561:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495925760:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.120813:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6581, 6581] 00000080:00200000:0.0:1789760781.120829:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.120836:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.120853:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26959872 i_blocks 53257 00000020:00200000:0.0:1789760781.120857:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.120866:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.120899:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26959872, count: 4096 00000080:00200000:0.0:1789760781.120906:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26959872, bytes: 4096 00000080:00200000:0.0:1789760781.120915:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.120924:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.120926:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26959872, 26963968) 0 0 00000080:00200000:0.0:1789760781.120930:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.120948:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6582, 6582] 00020000:00200000:0.0:1789760781.120957:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26959872 chunk: [26959872, 26963968) 26963968, 4096 00020000:00200000:0.0:1789760781.120962:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.120981:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6582, 6582] 00000008:00010000:0.0:1789760781.121000:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.121006:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.121018:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26959872, 26963968) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.121032:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26959872=0x19b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.121086:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.121089:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26963968 00000020:00010000:0.0:1789760781.121097:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.121101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.121104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.121108:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.121111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.121119:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.121125:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.121140:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760781.121391:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 26959872<>26963968 00000100:00100000:1.0:1789760781.121450:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495925888:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.149401:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973899 gen 2 00000008:00010000:1.0:1789760781.149452:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760781.149461:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.149473:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.149556:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760781.149586:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495925888:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.152806:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6582, 6582] 00000080:00200000:2.0:1789760781.152833:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.152840:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.152893:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26963968 i_blocks 53257 00000020:00200000:2.0:1789760781.152899:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.152917:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.152954:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26963968, count: 4096 00000080:00200000:2.0:1789760781.152963:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26963968, bytes: 4096 00000080:00200000:2.0:1789760781.153111:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.153122:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.153125:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26963968, 26968064) 0 0 00000080:00200000:2.0:1789760781.153130:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.153148:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6583, 6583] 00020000:00200000:2.0:1789760781.153157:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26963968 chunk: [26963968, 26968064) 26968064, 4096 00020000:00200000:2.0:1789760781.153163:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.153203:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6583, 6583] 00000008:00010000:2.0:1789760781.153223:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.153229:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.153240:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26963968, 26968064) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.153254:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26963968=0x19b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.153320:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.153323:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26968064 00000020:00010000:2.0:1789760781.153331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.153335:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.153339:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.153343:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.153346:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.153354:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.153362:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.153378:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760781.153507:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 26963968<>26968064 00000100:00100000:3.0:1789760781.153608:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495926016:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760781.179836:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973900 gen 2 00000008:00010000:3.0:1789760781.179887:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760781.179896:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.179907:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.179931:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760781.179961:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495926016:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.180297:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6583, 6583] 00000080:00200000:2.0:1789760781.180326:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.180332:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.180348:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26968064 i_blocks 53257 00000020:00200000:2.0:1789760781.180352:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.180361:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.180395:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26968064, count: 4096 00000080:00200000:2.0:1789760781.180401:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26968064, bytes: 4096 00000080:00200000:2.0:1789760781.180414:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.180422:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.180425:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26968064, 26972160) 0 0 00000080:00200000:2.0:1789760781.180429:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.180459:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6584, 6584] 00020000:00200000:2.0:1789760781.180468:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26968064 chunk: [26968064, 26972160) 26972160, 4096 00020000:00200000:2.0:1789760781.180473:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.180494:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6584, 6584] 00000008:00010000:2.0:1789760781.180512:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.180517:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.180529:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26968064, 26972160) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.180544:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26968064=0x19b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.180605:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.180608:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26972160 00000020:00010000:2.0:1789760781.180616:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.180619:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.180622:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.180625:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.180628:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.180822:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.180831:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.180849:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760781.181412:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 26968064<>26972160 00000100:00100000:3.0:1789760781.181546:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495926144:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760781.216999:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760781.217013:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.217024:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.217052:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760781.217081:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495926144:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.217204:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6584, 6584] 00000080:00200000:2.0:1789760781.217228:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.217235:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.217250:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26972160 i_blocks 53257 00000020:00200000:2.0:1789760781.217253:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.217262:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.217296:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26972160, count: 4096 00000080:00200000:2.0:1789760781.217303:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26972160, bytes: 4096 00000080:00200000:2.0:1789760781.217320:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.217329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.217331:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26972160, 26976256) 0 0 00000080:00200000:2.0:1789760781.217336:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.217352:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6585, 6585] 00020000:00200000:2.0:1789760781.217359:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26972160 chunk: [26972160, 26976256) 26976256, 4096 00020000:00200000:2.0:1789760781.217364:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.217382:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6585, 6585] 00000008:00010000:2.0:1789760781.217609:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.217616:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.217628:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26972160, 26976256) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.217642:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26972160=0x19b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.217697:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.217700:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26976256 00000020:00010000:2.0:1789760781.217708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.217711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.217714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.217718:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.217721:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.217728:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.217734:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.217749:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760781.217863:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 26972160<>26976256 00000100:00100000:3.0:1789760781.217972:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495926272:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760781.257781:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973902 gen 2 00000008:00010000:3.0:1789760781.257846:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760781.257945:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.257957:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.258037:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760781.258065:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495926272:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.258133:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6585, 6585] 00000080:00200000:2.0:1789760781.258152:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.258158:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.258202:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26976256 i_blocks 53257 00000020:00200000:2.0:1789760781.258206:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.258217:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.258251:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26976256, count: 4096 00000080:00200000:2.0:1789760781.258258:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26976256, bytes: 4096 00000080:00200000:2.0:1789760781.258268:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.258275:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.258277:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26976256, 26980352) 0 0 00000080:00200000:2.0:1789760781.258281:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.258298:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6586, 6586] 00020000:00200000:2.0:1789760781.258306:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26976256 chunk: [26976256, 26980352) 26980352, 4096 00020000:00200000:2.0:1789760781.258321:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.258343:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6586, 6586] 00000008:00010000:2.0:1789760781.258361:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.258367:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.258377:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26976256, 26980352) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.258391:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26976256=0x19ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.258442:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.258445:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26980352 00000020:00010000:2.0:1789760781.258452:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.258454:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.258457:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.258459:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.258461:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.258466:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.258471:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.258483:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760781.258995:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 26976256<>26980352 00000100:00100000:3.0:1789760781.259055:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760781.259074:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495926400:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760781.284013:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973903 gen 2 00000008:00010000:2.0:1789760781.284069:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760781.284077:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760781.284086:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760781.284107:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760781.284130:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495926400:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760781.284245:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6586, 6586] 00000080:00200000:1.0:1789760781.284264:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760781.284271:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760781.284286:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26980352 i_blocks 53257 00000020:00200000:1.0:1789760781.284292:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760781.284301:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760781.284342:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26980352, count: 4096 00000080:00200000:1.0:1789760781.284357:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26980352, bytes: 4096 00000080:00200000:1.0:1789760781.284368:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760781.284377:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760781.284380:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26980352, 26984448) 0 0 00000080:00200000:1.0:1789760781.284384:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760781.284398:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6587, 6587] 00020000:00200000:1.0:1789760781.284406:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26980352 chunk: [26980352, 26984448) 26984448, 4096 00020000:00200000:1.0:1789760781.284410:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760781.284427:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6587, 6587] 00000008:00010000:1.0:1789760781.284444:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760781.284449:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760781.284459:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26980352, 26984448) append 0 append lockpos 0 00000080:00200000:1.0:1789760781.284472:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26980352=0x19bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760781.284529:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760781.284532:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26984448 00000020:00010000:1.0:1789760781.284540:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760781.284544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.284547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.284550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760781.284553:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760781.284560:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760781.284567:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760781.284581:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760781.284716:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 26980352<>26984448 00000100:00100000:1.0:1789760781.284754:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760781.284773:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495926528:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.328559:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973904 gen 2 00000008:00010000:1.0:1789760781.328620:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760781.328629:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.328641:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.328659:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760781.328688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495926528:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.329138:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6587, 6587] 00000080:00200000:0.0:1789760781.329155:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.329162:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.329206:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26984448 i_blocks 53257 00000020:00200000:0.0:1789760781.329210:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.329220:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.329252:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26984448, count: 4096 00000080:00200000:0.0:1789760781.329258:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26984448, bytes: 4096 00000080:00200000:0.0:1789760781.329269:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.329278:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.329281:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26984448, 26988544) 0 0 00000080:00200000:0.0:1789760781.329285:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.329301:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6588, 6588] 00020000:00200000:0.0:1789760781.329320:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26984448 chunk: [26984448, 26988544) 26988544, 4096 00020000:00200000:0.0:1789760781.329325:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.329345:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6588, 6588] 00000008:00010000:0.0:1789760781.329362:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.329369:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.329391:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26984448, 26988544) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.329405:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26984448=0x19bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.329471:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.329474:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26988544 00000020:00010000:0.0:1789760781.329483:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.329487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.329490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.329494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.329497:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.329505:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.329512:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.329678:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760781.329810:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 26984448<>26988544 00000100:00100000:1.0:1789760781.329890:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495926656:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.375365:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973905 gen 2 00000008:00010000:1.0:1789760781.375421:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760781.375430:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.375443:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.375565:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760781.375600:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495926656:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.375811:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6588, 6588] 00000080:00200000:0.0:1789760781.375830:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.375847:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.375867:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26988544 i_blocks 53257 00000020:00200000:0.0:1789760781.375872:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.375882:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.375925:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26988544, count: 4096 00000080:00200000:0.0:1789760781.375933:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26988544, bytes: 4096 00000080:00200000:0.0:1789760781.375960:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.375969:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.375972:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26988544, 26992640) 0 0 00000080:00200000:0.0:1789760781.375976:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.375994:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6589, 6589] 00020000:00200000:0.0:1789760781.376361:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26988544 chunk: [26988544, 26992640) 26992640, 4096 00020000:00200000:0.0:1789760781.376368:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.376391:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6589, 6589] 00000008:00010000:0.0:1789760781.376412:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.376418:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.376431:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26988544, 26992640) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.376446:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26988544=0x19bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.376510:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.376513:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26992640 00000020:00010000:0.0:1789760781.376522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.376526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.376529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.376533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.376536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.376546:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.376553:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.376577:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760781.376752:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26988544<>26992640 00000100:00100000:1.0:1789760781.377075:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495926784:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.416491:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973906 gen 2 00000008:00010000:0.0:1789760781.416613:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:0.0:1789760781.416623:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760781.416632:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760781.416655:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:0.0:1789760781.416684:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495926784:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.416869:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6589, 6589] 00000080:00200000:2.0:1789760781.416887:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.416893:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.416909:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26992640 i_blocks 53257 00000020:00200000:2.0:1789760781.416912:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.416922:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.416958:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26992640, count: 4096 00000080:00200000:2.0:1789760781.416974:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26992640, bytes: 4096 00000080:00200000:2.0:1789760781.416985:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.416994:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.416997:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26992640, 26996736) 0 0 00000080:00200000:2.0:1789760781.417001:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.417018:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6590, 6590] 00020000:00200000:2.0:1789760781.417026:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26992640 chunk: [26992640, 26996736) 26996736, 4096 00020000:00200000:2.0:1789760781.417031:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.417061:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6590, 6590] 00000008:00010000:2.0:1789760781.417083:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.417089:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.417101:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26992640, 26996736) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.417116:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26992640=0x19be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.417198:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.417201:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26996736 00000020:00010000:2.0:1789760781.417210:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.417213:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.417216:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.417219:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.417222:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.417229:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.417237:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.417253:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760781.417385:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 26992640<>26996736 00000100:00100000:2.0:1789760781.417436:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760781.417453:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495926912:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760781.483477:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973907 gen 2 00000008:00010000:2.0:1789760781.483606:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760781.483615:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760781.483627:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760781.483657:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760781.483689:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495926912:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.484386:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6590, 6590] 00000080:00200000:0.0:1789760781.484567:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.484575:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.484593:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26996736 i_blocks 53257 00000020:00200000:0.0:1789760781.484597:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.484608:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.484650:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26996736, count: 4096 00000080:00200000:0.0:1789760781.484671:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26996736, bytes: 4096 00000080:00200000:0.0:1789760781.484686:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.484697:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.484938:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26996736, 27000832) 0 0 00000080:00200000:0.0:1789760781.484946:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.484964:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6591, 6591] 00020000:00200000:0.0:1789760781.484973:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26996736 chunk: [26996736, 27000832) 27000832, 4096 00020000:00200000:0.0:1789760781.484978:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.485001:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6591, 6591] 00000008:00010000:0.0:1789760781.485019:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.485026:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.485038:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26996736, 27000832) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.485052:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=26996736=0x19bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.485121:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.485125:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27000832 00000020:00010000:0.0:1789760781.485134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.485138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.485141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.485145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.485148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.485161:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.485783:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.485814:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760781.485963:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 26996736<>27000832 00000100:00100000:1.0:1789760781.486074:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927040:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.534144:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973908 gen 2 00000008:00010000:1.0:1789760781.534224:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760781.534233:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.534244:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.534320:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760781.534351:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927040:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.534484:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6591, 6591] 00000080:00200000:0.0:1789760781.534530:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.534537:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.534552:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27000832 i_blocks 53257 00000020:00200000:0.0:1789760781.534556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.534565:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.534604:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27000832, count: 4096 00000080:00200000:0.0:1789760781.534611:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27000832, bytes: 4096 00000080:00200000:0.0:1789760781.534621:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.534629:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.534632:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27000832, 27004928) 0 0 00000080:00200000:0.0:1789760781.534637:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.534655:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6592, 6592] 00020000:00200000:0.0:1789760781.534663:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27000832 chunk: [27000832, 27004928) 27004928, 4096 00020000:00200000:0.0:1789760781.534668:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.534687:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6592, 6592] 00000008:00010000:0.0:1789760781.534715:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.534721:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.534732:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27000832, 27004928) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.534747:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27000832=0x19c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.535025:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.535028:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27004928 00000020:00010000:0.0:1789760781.535037:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.535041:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.535044:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.535048:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.535050:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.535058:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.535064:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.535083:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760781.535365:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 27000832<>27004928 00000100:00100000:1.0:1789760781.535419:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760781.535441:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927168:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760781.574808:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760781.574837:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.575395:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.575488:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760781.575521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927168:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.575758:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6592, 6592] 00000080:00200000:2.0:1789760781.575776:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.575782:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.575799:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27004928 i_blocks 53257 00000020:00200000:2.0:1789760781.575803:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.575820:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.576085:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27004928, count: 4096 00000080:00200000:2.0:1789760781.576094:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27004928, bytes: 4096 00000080:00200000:2.0:1789760781.576108:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.576117:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.576119:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27004928, 27009024) 0 0 00000080:00200000:2.0:1789760781.576124:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.576141:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6593, 6593] 00020000:00200000:2.0:1789760781.576149:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27004928 chunk: [27004928, 27009024) 27009024, 4096 00020000:00200000:2.0:1789760781.576154:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.576202:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6593, 6593] 00000008:00010000:2.0:1789760781.576220:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.576226:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.576237:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27004928, 27009024) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.576251:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27004928=0x19c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.576389:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.576393:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27009024 00000020:00010000:2.0:1789760781.576402:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.576406:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.576409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.576413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.576416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.576424:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.576430:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.576451:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760781.576575:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 27004928<>27009024 00000100:00100000:3.0:1789760781.576664:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495927296:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760781.611798:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973910 gen 2 00000008:00010000:2.0:1789760781.611869:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760781.611877:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760781.611888:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760781.611912:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760781.611940:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495927296:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.612037:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6593, 6593] 00000080:00200000:0.0:1789760781.612055:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.612061:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.612077:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27009024 i_blocks 53257 00000020:00200000:0.0:1789760781.612080:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.612090:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.612123:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27009024, count: 4096 00000080:00200000:0.0:1789760781.612130:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27009024, bytes: 4096 00000080:00200000:0.0:1789760781.612140:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.612149:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.612151:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27009024, 27013120) 0 0 00000080:00200000:0.0:1789760781.612162:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.612238:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6594, 6594] 00020000:00200000:0.0:1789760781.612246:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27009024 chunk: [27009024, 27013120) 27013120, 4096 00020000:00200000:0.0:1789760781.612255:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.612276:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6594, 6594] 00000008:00010000:0.0:1789760781.612294:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.612299:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.612320:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27009024, 27013120) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.612335:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27009024=0x19c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.612390:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.612638:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27013120 00000020:00010000:0.0:1789760781.612649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.612653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.612656:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.612659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.612662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.612670:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.612678:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.612694:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760781.612816:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 27009024<>27013120 00000100:00100000:1.0:1789760781.612905:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927424:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760781.653949:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760781.653966:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.653977:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.654824:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760781.654862:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927424:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.654947:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6594, 6594] 00000080:00200000:0.0:1789760781.654964:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.654971:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.654986:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27013120 i_blocks 53257 00000020:00200000:0.0:1789760781.654990:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.655000:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.655045:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27013120, count: 4096 00000080:00200000:0.0:1789760781.655053:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27013120, bytes: 4096 00000080:00200000:0.0:1789760781.655066:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.655075:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.655078:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27013120, 27017216) 0 0 00000080:00200000:0.0:1789760781.655083:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.655100:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6595, 6595] 00020000:00200000:0.0:1789760781.655108:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27013120 chunk: [27013120, 27017216) 27017216, 4096 00020000:00200000:0.0:1789760781.655113:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.655135:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6595, 6595] 00000008:00010000:0.0:1789760781.655153:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.655159:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.655185:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27013120, 27017216) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.655201:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27013120=0x19c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.655257:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.655259:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27017216 00000020:00010000:0.0:1789760781.655268:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.655272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.655275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.655279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.655282:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.655289:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.655295:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.655325:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760781.655461:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 27013120<>27017216 00000100:00100000:1.0:1789760781.655584:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495927552:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.694825:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973912 gen 2 00000008:00010000:1.0:1789760781.694892:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760781.694900:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.694910:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.694975:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760781.695014:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495927552:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.695467:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6595, 6595] 00000080:00200000:0.0:1789760781.695485:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.695492:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.695507:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27017216 i_blocks 53257 00000020:00200000:0.0:1789760781.695511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.695520:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.695550:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27017216, count: 4096 00000080:00200000:0.0:1789760781.695558:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27017216, bytes: 4096 00000080:00200000:0.0:1789760781.695569:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.695578:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.695580:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27017216, 27021312) 0 0 00000080:00200000:0.0:1789760781.695586:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.695601:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6596, 6596] 00020000:00200000:0.0:1789760781.695609:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27017216 chunk: [27017216, 27021312) 27021312, 4096 00020000:00200000:0.0:1789760781.695614:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.695685:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6596, 6596] 00000008:00010000:0.0:1789760781.695754:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.695761:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760781.695772:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27017216, 27021312) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.695786:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27017216=0x19c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.695836:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.695838:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27021312 00000020:00010000:0.0:1789760781.695846:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.695849:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.695852:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.695855:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.695858:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.695865:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.695870:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.695886:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760781.696056:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 27017216<>27021312 00000100:00100000:1.0:1789760781.696120:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927680:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.731540:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973913 gen 2 00000008:00010000:1.0:1789760781.731611:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760781.731619:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.731639:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.731661:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760781.731691:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927680:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760781.731745:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6596, 6596] 00000080:00200000:0.0:1789760781.731770:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.731777:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760781.731891:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27021312 i_blocks 53257 00000020:00200000:1.0:1789760781.731896:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760781.731904:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760781.731936:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27021312, count: 4096 00000080:00200000:1.0:1789760781.731943:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27021312, bytes: 4096 00000080:00200000:1.0:1789760781.731954:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760781.731963:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760781.731966:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27021312, 27025408) 0 0 00000080:00200000:1.0:1789760781.731971:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760781.731988:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6597, 6597] 00020000:00200000:1.0:1789760781.731997:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27021312 chunk: [27021312, 27025408) 27025408, 4096 00020000:00200000:1.0:1789760781.732002:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760781.732022:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6597, 6597] 00000008:00010000:1.0:1789760781.732041:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760781.732046:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760781.732058:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27021312, 27025408) append 0 append lockpos 0 00000080:00200000:1.0:1789760781.732072:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27021312=0x19c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760781.732238:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760781.732241:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27025408 00000020:00010000:1.0:1789760781.732250:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760781.732254:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.732257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.732260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760781.732263:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760781.732270:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760781.732276:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760781.732292:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760781.732432:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27021312<>27025408 00000100:00100000:1.0:1789760781.732476:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760781.732492:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927808:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.759375:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973914 gen 2 00000008:00010000:1.0:1789760781.759429:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760781.759438:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.759449:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.759483:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760781.759517:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495927808:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.759727:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6597, 6597] 00000080:00200000:2.0:1789760781.759744:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.759750:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.759766:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27025408 i_blocks 53257 00000020:00200000:2.0:1789760781.759769:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.759779:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.759814:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27025408, count: 4096 00000080:00200000:2.0:1789760781.759822:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27025408, bytes: 4096 00000080:00200000:2.0:1789760781.759837:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.759845:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.759847:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27025408, 27029504) 0 0 00000080:00200000:2.0:1789760781.759852:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.759869:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6598, 6598] 00020000:00200000:2.0:1789760781.759877:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27025408 chunk: [27025408, 27029504) 27029504, 4096 00020000:00200000:2.0:1789760781.759882:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.759901:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6598, 6598] 00000008:00010000:2.0:1789760781.759920:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.759926:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.759938:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27025408, 27029504) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.759951:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27025408=0x19c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.760002:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.760005:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27029504 00000020:00010000:2.0:1789760781.760013:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.760016:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.760020:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.760023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.760026:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.760033:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.760039:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.760054:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760781.760336:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 27025408<>27029504 00000100:00100000:2.0:1789760781.760393:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495927936:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760781.771478:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760781.814606:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973915 gen 2 00000008:00010000:2.0:1789760781.814659:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760781.814667:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760781.814679:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760781.814864:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760781.814896:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495927936:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760781.816372:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6598, 6598] 00000080:00200000:1.0:1789760781.816459:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760781.816467:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760781.816487:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27029504 i_blocks 53257 00000020:00200000:1.0:1789760781.816493:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760781.816509:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760781.816555:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27029504, count: 4096 00000080:00200000:1.0:1789760781.816563:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27029504, bytes: 4096 00000080:00200000:1.0:1789760781.816580:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760781.816590:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760781.816592:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27029504, 27033600) 0 0 00000080:00200000:1.0:1789760781.816598:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760781.816617:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6599, 6599] 00020000:00200000:1.0:1789760781.816626:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27029504 chunk: [27029504, 27033600) 27033600, 4096 00020000:00200000:1.0:1789760781.816631:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760781.816654:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6599, 6599] 00000008:00010000:1.0:1789760781.816673:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760781.816678:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760781.816690:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27029504, 27033600) append 0 append lockpos 0 00000080:00200000:1.0:1789760781.816911:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27029504=0x19c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760781.816984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760781.816987:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27033600 00000020:00010000:1.0:1789760781.816996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760781.817000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.817003:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.817007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760781.817010:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760781.817017:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760781.817025:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760781.817057:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760781.817289:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27029504<>27033600 00000100:00100000:0.0:1789760781.817391:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760781.817625:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495928064:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.874804:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973916 gen 2 00000008:00010000:1.0:1789760781.874857:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760781.874866:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.874878:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.874903:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760781.874931:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495928064:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.875146:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6599, 6599] 00000080:00200000:2.0:1789760781.875164:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.875195:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.875210:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27033600 i_blocks 53257 00000020:00200000:2.0:1789760781.875213:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.875223:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.875256:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27033600, count: 4096 00000080:00200000:2.0:1789760781.875262:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27033600, bytes: 4096 00000080:00200000:2.0:1789760781.875273:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.875281:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.875294:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27033600, 27037696) 0 0 00000080:00200000:2.0:1789760781.875299:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.875325:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6600, 6600] 00020000:00200000:2.0:1789760781.875335:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27033600 chunk: [27033600, 27037696) 27037696, 4096 00020000:00200000:2.0:1789760781.875339:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.875360:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6600, 6600] 00000008:00010000:2.0:1789760781.875380:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.875386:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.875398:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27033600, 27037696) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.875413:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27033600=0x19c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.875617:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.875621:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27037696 00000020:00010000:2.0:1789760781.875700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.875704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.875707:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.875711:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.875714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.875722:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.875730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.875751:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760781.875932:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 27033600<>27037696 00000100:00100000:3.0:1789760781.876009:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495928192:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760781.906788:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760781.906803:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.906815:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.906841:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760781.906879:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495928192:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.907131:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6600, 6600] 00000080:00200000:2.0:1789760781.907147:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.907154:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.907197:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27037696 i_blocks 53257 00000020:00200000:2.0:1789760781.907201:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.907211:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.907243:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27037696, count: 4096 00000080:00200000:2.0:1789760781.907250:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27037696, bytes: 4096 00000080:00200000:2.0:1789760781.907260:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.907268:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.907270:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27037696, 27041792) 0 0 00000080:00200000:2.0:1789760781.907275:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.907299:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6601, 6601] 00020000:00200000:2.0:1789760781.907318:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27037696 chunk: [27037696, 27041792) 27041792, 4096 00020000:00200000:2.0:1789760781.907323:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.907343:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6601, 6601] 00000008:00010000:2.0:1789760781.907360:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.907366:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.907377:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27037696, 27041792) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.907389:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27037696=0x19c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.907560:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.907563:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27041792 00000020:00010000:2.0:1789760781.907571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.907575:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.907578:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.907581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.907585:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.907592:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.907599:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.907617:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760781.907789:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 27037696<>27041792 00000100:00100000:2.0:1789760781.907838:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495928320:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760781.934051:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973918 gen 2 00000008:00010000:2.0:1789760781.934115:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760781.934124:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760781.934135:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760781.934248:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760781.934275:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495928320:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760781.934353:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6601, 6601] 00000080:00200000:1.0:1789760781.934370:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760781.934383:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760781.934398:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27041792 i_blocks 53257 00000020:00200000:1.0:1789760781.934402:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760781.934412:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760781.934446:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27041792, count: 4096 00000080:00200000:1.0:1789760781.934453:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27041792, bytes: 4096 00000080:00200000:1.0:1789760781.934463:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760781.934472:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760781.934474:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27041792, 27045888) 0 0 00000080:00200000:1.0:1789760781.934479:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760781.934495:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6602, 6602] 00020000:00200000:1.0:1789760781.934503:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27041792 chunk: [27041792, 27045888) 27045888, 4096 00020000:00200000:1.0:1789760781.934507:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760781.934525:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6602, 6602] 00000008:00010000:1.0:1789760781.934542:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760781.934547:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760781.934557:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27041792, 27045888) append 0 append lockpos 0 00000080:00200000:1.0:1789760781.934570:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27041792=0x19ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760781.934616:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760781.934618:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27045888 00000020:00010000:1.0:1789760781.934626:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760781.934630:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.934633:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.934637:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760781.934639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760781.934645:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760781.934652:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760781.934666:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760781.934789:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27041792<>27045888 00000100:00100000:0.0:1789760781.934861:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760781.936371:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495928448:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760781.961860:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973919 gen 2 00000008:00010000:1.0:1789760781.961913:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760781.961922:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.961933:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.961954:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760781.961983:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495928448:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760781.961993:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6602, 6602] 00000080:00200000:2.0:1789760781.962013:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.962020:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.962037:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27045888 i_blocks 53257 00000020:00200000:2.0:1789760781.962041:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.962053:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.962143:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27045888, count: 4096 00000080:00200000:2.0:1789760781.962152:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27045888, bytes: 4096 00000080:00200000:2.0:1789760781.962162:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.962199:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.962201:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27045888, 27049984) 0 0 00000080:00200000:2.0:1789760781.962206:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.962221:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6603, 6603] 00020000:00200000:2.0:1789760781.962229:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27045888 chunk: [27045888, 27049984) 27049984, 4096 00020000:00200000:2.0:1789760781.962233:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.962252:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6603, 6603] 00000008:00010000:2.0:1789760781.962347:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.962354:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760781.962366:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27045888, 27049984) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.962379:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27045888=0x19cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.962430:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.962433:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27049984 00000020:00010000:2.0:1789760781.962450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.962454:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.962456:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.962460:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.962463:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.962471:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.962482:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.962500:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760781.962619:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 27045888<>27049984 00000100:00100000:2.0:1789760781.962790:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760781.962808:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495928576:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760782.014068:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973920 gen 2 00000008:00010000:2.0:1789760782.014122:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760782.014130:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.014142:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.014213:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760782.014243:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495928576:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760782.014813:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6603, 6603] 00000080:00200000:1.0:1789760782.014908:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760782.014916:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760782.014932:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27049984 i_blocks 53257 00000020:00200000:1.0:1789760782.014936:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760782.014947:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760782.014984:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27049984, count: 4096 00000080:00200000:1.0:1789760782.014991:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27049984, bytes: 4096 00000080:00200000:1.0:1789760782.015005:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760782.015013:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760782.015016:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27049984, 27054080) 0 0 00000080:00200000:1.0:1789760782.015021:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760782.015040:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6604, 6604] 00020000:00200000:1.0:1789760782.015050:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27049984 chunk: [27049984, 27054080) 27054080, 4096 00020000:00200000:1.0:1789760782.015054:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760782.015077:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6604, 6604] 00000008:00010000:1.0:1789760782.015099:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760782.015104:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760782.015116:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27049984, 27054080) append 0 append lockpos 0 00000080:00200000:1.0:1789760782.015134:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27049984=0x19cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760782.015225:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760782.015228:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27054080 00000020:00010000:1.0:1789760782.015237:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760782.015240:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760782.015243:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760782.015246:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760782.015249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760782.015256:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760782.015263:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760782.015285:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760782.015421:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27049984<>27054080 00000100:00100000:0.0:1789760782.016981:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495928704:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.058934:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973921 gen 2 00000008:00010000:1.0:1789760782.058988:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760782.058997:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.059009:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.059047:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760782.059078:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495928704:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760782.059145:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6604, 6604] 00000080:00200000:2.0:1789760782.059163:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760782.059201:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760782.059219:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27054080 i_blocks 53257 00000020:00200000:2.0:1789760782.059224:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760782.059234:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760782.059319:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27054080, count: 4096 00000080:00200000:2.0:1789760782.059328:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27054080, bytes: 4096 00000080:00200000:2.0:1789760782.059339:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760782.059347:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760782.059349:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27054080, 27058176) 0 0 00000080:00200000:2.0:1789760782.059353:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760782.059369:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6605, 6605] 00020000:00200000:2.0:1789760782.059377:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27054080 chunk: [27054080, 27058176) 27058176, 4096 00020000:00200000:2.0:1789760782.059382:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760782.059401:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6605, 6605] 00000008:00010000:2.0:1789760782.059417:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760782.059423:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760782.059434:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27054080, 27058176) append 0 append lockpos 0 00000080:00200000:2.0:1789760782.059449:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27054080=0x19cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760782.059506:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760782.059509:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27058176 00000020:00010000:2.0:1789760782.059519:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760782.059522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.059526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.059529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760782.059532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760782.059540:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760782.059547:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760782.059564:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760782.059689:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 27054080<>27058176 00000100:00100000:3.0:1789760782.059761:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760782.060134:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495928832:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760782.109314:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973922 gen 2 00000008:00010000:3.0:1789760782.109372:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760782.109381:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.109392:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.109414:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760782.109443:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495928832:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760782.109801:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6605, 6605] 00000080:00200000:2.0:1789760782.109820:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760782.109827:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760782.109843:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27058176 i_blocks 53257 00000020:00200000:2.0:1789760782.109847:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760782.109857:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760782.109892:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27058176, count: 4096 00000080:00200000:2.0:1789760782.109899:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27058176, bytes: 4096 00000080:00200000:2.0:1789760782.109909:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760782.109917:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760782.109919:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27058176, 27062272) 0 0 00000080:00200000:2.0:1789760782.109924:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760782.109940:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6606, 6606] 00020000:00200000:2.0:1789760782.109948:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27058176 chunk: [27058176, 27062272) 27062272, 4096 00020000:00200000:2.0:1789760782.109954:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760782.109974:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6606, 6606] 00000008:00010000:2.0:1789760782.109992:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760782.109998:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760782.110010:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27058176, 27062272) append 0 append lockpos 0 00000080:00200000:2.0:1789760782.110024:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27058176=0x19ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760782.110083:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760782.110086:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27062272 00000020:00010000:2.0:1789760782.110095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760782.110098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.110101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.110105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760782.110107:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760782.110115:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760782.110122:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760782.110139:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760782.110295:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 27058176<>27062272 00000100:00100000:2.0:1789760782.110349:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760782.110366:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495928960:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760782.175380:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973923 gen 2 00000008:00010000:2.0:1789760782.175444:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760782.175452:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.175463:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.175491:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760782.175518:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495928960:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760782.177990:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6606, 6606] 00000080:00200000:3.0:1789760782.178055:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.178062:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.178081:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27062272 i_blocks 53257 00000020:00200000:3.0:1789760782.178086:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.178097:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.178142:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27062272, count: 4096 00000080:00200000:3.0:1789760782.178151:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27062272, bytes: 4096 00000080:00200000:3.0:1789760782.178166:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.178198:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.179310:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27062272, 27066368) 0 0 00000080:00200000:3.0:1789760782.179318:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.179339:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6607, 6607] 00020000:00200000:3.0:1789760782.179420:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27062272 chunk: [27062272, 27066368) 27066368, 4096 00020000:00200000:3.0:1789760782.179426:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.179455:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6607, 6607] 00000008:00010000:3.0:1789760782.179475:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.179481:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760782.179493:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27062272, 27066368) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.179508:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27062272=0x19cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.179581:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.179584:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27066368 00000020:00010000:3.0:1789760782.179593:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.179597:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.179600:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.179604:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.179787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.179802:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.179812:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.179844:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760782.180074:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 27062272<>27066368 00000100:00100000:2.0:1789760782.180166:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495929088:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760782.254491:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973924 gen 2 00000008:00010000:2.0:1789760782.254548:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:2.0:1789760782.254559:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.254572:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.257799:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:2.0:1789760782.257856:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495929088:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760782.259159:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6607, 6607] 00000080:00200000:3.0:1789760782.259196:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.259204:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.259233:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27066368 i_blocks 53257 00000020:00200000:3.0:1789760782.259239:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.259449:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.259486:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27066368, count: 4096 00000080:00200000:3.0:1789760782.259495:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27066368, bytes: 4096 00000080:00200000:3.0:1789760782.259511:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.259520:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.259523:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27066368, 27070464) 0 0 00000080:00200000:3.0:1789760782.259528:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.259546:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6608, 6608] 00020000:00200000:3.0:1789760782.259555:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27066368 chunk: [27066368, 27070464) 27070464, 4096 00020000:00200000:3.0:1789760782.259560:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.259583:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6608, 6608] 00000008:00010000:3.0:1789760782.259601:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.259607:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760782.259620:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27066368, 27070464) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.259634:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27066368=0x19d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.259873:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.259876:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27070464 00000020:00010000:3.0:1789760782.259886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.259889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.259892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.259896:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.259899:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.259906:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.259914:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.259934:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760782.260064:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 27066368<>27070464 00000100:00100000:3.0:1789760782.260133:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760782.260156:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495929216:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760782.329817:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973925 gen 2 00000008:00010000:2.0:1789760782.330024:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:2.0:1789760782.330034:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.330046:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.330105:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:2.0:1789760782.330137:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495929216:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760782.330264:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6608, 6608] 00000080:00200000:3.0:1789760782.333453:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.333471:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760782.335145:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27070464 i_blocks 53257 00000020:00200000:2.0:1789760782.335158:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760782.335208:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760782.335247:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27070464, count: 4096 00000080:00200000:2.0:1789760782.335257:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27070464, bytes: 4096 00000080:00200000:2.0:1789760782.335273:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760782.335283:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760782.335286:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27070464, 27074560) 0 0 00000080:00200000:2.0:1789760782.335291:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760782.335328:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6609, 6609] 00020000:00200000:2.0:1789760782.335339:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27070464 chunk: [27070464, 27074560) 27074560, 4096 00020000:00200000:2.0:1789760782.335344:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760782.335367:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6609, 6609] 00000008:00010000:2.0:1789760782.335389:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760782.335395:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760782.335407:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27070464, 27074560) append 0 append lockpos 0 00000080:00200000:2.0:1789760782.335423:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27070464=0x19d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760782.335490:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760782.335493:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27074560 00000020:00010000:2.0:1789760782.335501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760782.335505:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.335508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.335512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760782.335515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760782.335523:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760782.335531:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760782.335592:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760782.335724:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 27070464<>27074560 00000100:00100000:2.0:1789760782.335797:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495929344:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760782.373518:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973926 gen 2 00000008:00010000:2.0:1789760782.373568:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760782.373577:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.373587:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.373618:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000080:00200000:0.0:1789760782.373709:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6609, 6609] 00000080:00200000:0.0:1789760782.373728:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.373734:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.373749:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27074560 i_blocks 53257 00000020:00200000:0.0:1789760782.373752:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.373761:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.373794:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27074560, count: 4096 00000080:00200000:0.0:1789760782.373803:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27074560, bytes: 4096 00000080:00200000:0.0:1789760782.373812:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.373821:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.373824:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27074560, 27078656) 0 0 00000080:00200000:0.0:1789760782.373829:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.373845:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6610, 6610] 00020000:00200000:0.0:1789760782.373863:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27074560 chunk: [27074560, 27078656) 27078656, 4096 00020000:00200000:0.0:1789760782.373868:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.373888:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6610, 6610] 00000008:00010000:0.0:1789760782.373906:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.373912:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.373923:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27074560, 27078656) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.373937:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27074560=0x19d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.374000:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.374003:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27078656 00000020:00010000:0.0:1789760782.374011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.374015:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.374018:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.374022:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.374025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.374032:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.374039:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.374058:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760782.374226:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27074560<>27078656 00000100:00100000:2.0:1789760782.374355:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495929344:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.374572:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495929472:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760782.405915:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760782.405934:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.405944:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.406006:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760782.406039:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495929472:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.406161:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6610, 6610] 00000080:00200000:0.0:1789760782.406209:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.406216:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.406230:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27078656 i_blocks 53257 00000020:00200000:0.0:1789760782.406234:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.406244:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.406277:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27078656, count: 4096 00000080:00200000:0.0:1789760782.406453:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27078656, bytes: 4096 00000080:00200000:0.0:1789760782.406464:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.406472:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.406475:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27078656, 27082752) 0 0 00000080:00200000:0.0:1789760782.406479:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.406496:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6611, 6611] 00020000:00200000:0.0:1789760782.406505:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27078656 chunk: [27078656, 27082752) 27082752, 4096 00020000:00200000:0.0:1789760782.406510:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.406530:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6611, 6611] 00000008:00010000:0.0:1789760782.406547:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.406552:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.406563:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27078656, 27082752) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.406577:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27078656=0x19d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.406630:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.406633:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27082752 00000020:00010000:0.0:1789760782.406642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.406646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.406649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.406652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.406655:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.406663:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.406670:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.406688:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760782.406809:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 27078656<>27082752 00000100:00100000:1.0:1789760782.406884:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495929600:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.445527:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973927 gen 2 00000100:00100000:1.0:1789760782.445567:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211837b80 x1876699495929600/t4294973928(4294973928) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760798 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760782.445614:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760782.445621:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.445631:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.445690:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760782.445713:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495929600:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.445779:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6611, 6611] 00000080:00200000:0.0:1789760782.445795:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.445801:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.445815:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27082752 i_blocks 53257 00000020:00200000:0.0:1789760782.445818:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.445827:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.446047:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27082752, count: 4096 00000080:00200000:0.0:1789760782.446055:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27082752, bytes: 4096 00000080:00200000:0.0:1789760782.446065:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.446074:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.446076:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27082752, 27086848) 0 0 00000080:00200000:0.0:1789760782.446080:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.446098:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6612, 6612] 00020000:00200000:0.0:1789760782.446106:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27082752 chunk: [27082752, 27086848) 27086848, 4096 00020000:00200000:0.0:1789760782.446110:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.446129:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6612, 6612] 00000008:00010000:0.0:1789760782.446144:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.446150:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.446162:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27082752, 27086848) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.446276:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27082752=0x19d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.446343:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.446346:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27086848 00000020:00010000:0.0:1789760782.446355:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.446359:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.446362:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.446365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.446368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.446376:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.446383:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.446401:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760782.446527:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 27082752<>27086848 00000100:00100000:1.0:1789760782.446614:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495929728:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.488685:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973929 gen 2 00000008:00010000:1.0:1789760782.488747:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760782.488756:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.488766:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.488830:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760782.488860:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495929728:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.489247:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6612, 6612] 00000080:00200000:0.0:1789760782.489436:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.489443:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.489460:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27086848 i_blocks 53257 00000020:00200000:0.0:1789760782.489464:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.489473:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.489507:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27086848, count: 4096 00000080:00200000:0.0:1789760782.489514:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27086848, bytes: 4096 00000080:00200000:0.0:1789760782.489524:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.489533:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.489535:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27086848, 27090944) 0 0 00000080:00200000:0.0:1789760782.489540:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.489554:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6613, 6613] 00020000:00200000:0.0:1789760782.489562:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27086848 chunk: [27086848, 27090944) 27090944, 4096 00020000:00200000:0.0:1789760782.489567:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.489587:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6613, 6613] 00000008:00010000:0.0:1789760782.489603:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.489609:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.489621:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27086848, 27090944) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.489636:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27086848=0x19d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.489695:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.489699:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27090944 00000020:00010000:0.0:1789760782.489706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.489710:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.489713:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.489716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.489719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.489726:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.489732:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.489751:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760782.490015:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 27086848<>27090944 00000100:00100000:1.0:1789760782.490086:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.490106:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495929856:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760782.524999:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760782.525017:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.525028:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.525091:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760782.525123:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495929856:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.525253:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6613, 6613] 00000080:00200000:0.0:1789760782.525272:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.525279:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.525359:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27090944 i_blocks 53257 00000020:00200000:0.0:1789760782.525364:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.525374:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.525410:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27090944, count: 4096 00000080:00200000:0.0:1789760782.525417:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27090944, bytes: 4096 00000080:00200000:0.0:1789760782.525428:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.525437:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.525439:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27090944, 27095040) 0 0 00000080:00200000:0.0:1789760782.525443:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.525460:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6614, 6614] 00020000:00200000:0.0:1789760782.525469:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27090944 chunk: [27090944, 27095040) 27095040, 4096 00020000:00200000:0.0:1789760782.525474:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.525496:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6614, 6614] 00000008:00010000:0.0:1789760782.525514:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.525520:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.525532:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27090944, 27095040) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.525546:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27090944=0x19d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.525600:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.525603:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27095040 00000020:00010000:0.0:1789760782.525611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.525615:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.525617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.525621:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.525624:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.525699:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.525707:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.525725:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760782.525927:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 27090944<>27095040 00000100:00100000:1.0:1789760782.525994:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495929984:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.559952:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973930 gen 2 00000100:00100000:1.0:1789760782.559996:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211837100 x1876699495929984/t4294973931(4294973931) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760798 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760782.560042:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760782.560050:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.560061:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.560083:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760782.560107:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495929984:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.560248:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6614, 6614] 00000080:00200000:0.0:1789760782.560265:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.560271:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.560357:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27095040 i_blocks 53257 00000020:00200000:0.0:1789760782.560362:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.560372:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.560407:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27095040, count: 4096 00000080:00200000:0.0:1789760782.560415:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27095040, bytes: 4096 00000080:00200000:0.0:1789760782.560424:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.560432:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.560434:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27095040, 27099136) 0 0 00000080:00200000:0.0:1789760782.560438:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.560455:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6615, 6615] 00020000:00200000:0.0:1789760782.560465:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27095040 chunk: [27095040, 27099136) 27099136, 4096 00020000:00200000:0.0:1789760782.560469:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.560489:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6615, 6615] 00000008:00010000:0.0:1789760782.560508:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.560514:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.560525:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27095040, 27099136) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.560539:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27095040=0x19d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.560596:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.560599:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27099136 00000020:00010000:0.0:1789760782.560609:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.560612:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.560616:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.560620:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.560694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.560703:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.560710:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.560727:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760782.560865:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 27095040<>27099136 00000100:00100000:1.0:1789760782.560938:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495930112:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.584985:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973932 gen 2 00000008:00010000:1.0:1789760782.585052:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760782.585060:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.585071:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.585091:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760782.585119:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495930112:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.585330:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6615, 6615] 00000080:00200000:0.0:1789760782.585347:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.585353:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.585369:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27099136 i_blocks 53257 00000020:00200000:0.0:1789760782.585372:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.585392:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.585427:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27099136, count: 4096 00000080:00200000:0.0:1789760782.585435:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27099136, bytes: 4096 00000080:00200000:0.0:1789760782.585444:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.585453:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.585455:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27099136, 27103232) 0 0 00000080:00200000:0.0:1789760782.585460:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.585475:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6616, 6616] 00020000:00200000:0.0:1789760782.585483:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27099136 chunk: [27099136, 27103232) 27103232, 4096 00020000:00200000:0.0:1789760782.585488:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.585508:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6616, 6616] 00000008:00010000:0.0:1789760782.585525:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.585530:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.585541:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27099136, 27103232) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.585556:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27099136=0x19d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.585616:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.585619:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27103232 00000020:00010000:0.0:1789760782.585640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.585644:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.585647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.585650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.585653:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.585660:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.585668:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.585684:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760782.585811:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27099136<>27103232 00000100:00100000:1.0:1789760782.586165:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495930240:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.621903:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973933 gen 2 00000008:00010000:1.0:1789760782.621954:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760782.621962:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.621973:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.622037:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760782.622071:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495930240:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.623195:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6616, 6616] 00000080:00200000:0.0:1789760782.623216:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.623221:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.623292:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27103232 i_blocks 53257 00000020:00200000:0.0:1789760782.623297:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.623319:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.623356:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27103232, count: 4096 00000080:00200000:0.0:1789760782.623363:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27103232, bytes: 4096 00000080:00200000:0.0:1789760782.623373:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.623381:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.623383:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27103232, 27107328) 0 0 00000080:00200000:0.0:1789760782.623386:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.623400:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6617, 6617] 00020000:00200000:0.0:1789760782.623407:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27103232 chunk: [27103232, 27107328) 27107328, 4096 00020000:00200000:0.0:1789760782.623411:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.623429:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6617, 6617] 00000008:00010000:0.0:1789760782.623442:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.623447:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.623456:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27103232, 27107328) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.623467:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27103232=0x19d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.623518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.623521:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27107328 00000020:00010000:0.0:1789760782.623527:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.623531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.623534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.623537:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.623539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.623546:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.623551:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.623575:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760782.623685:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 27103232<>27107328 00000100:00100000:1.0:1789760782.623778:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.623805:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495930368:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.672944:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973934 gen 2 00000008:00010000:1.0:1789760782.672991:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760782.672999:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.673010:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.673074:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760782.673115:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495930368:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.674108:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6617, 6617] 00000080:00200000:0.0:1789760782.674132:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.674138:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.674155:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27107328 i_blocks 53257 00000020:00200000:0.0:1789760782.674160:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.674192:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.674230:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27107328, count: 4096 00000080:00200000:0.0:1789760782.674246:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27107328, bytes: 4096 00000080:00200000:0.0:1789760782.674259:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.674268:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.674270:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27107328, 27111424) 0 0 00000080:00200000:0.0:1789760782.674274:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.674293:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6618, 6618] 00020000:00200000:0.0:1789760782.674310:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27107328 chunk: [27107328, 27111424) 27111424, 4096 00020000:00200000:0.0:1789760782.674315:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.674336:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6618, 6618] 00000008:00010000:0.0:1789760782.674364:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.674370:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.674380:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27107328, 27111424) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.674394:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27107328=0x19da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.674458:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.674461:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27111424 00000020:00010000:0.0:1789760782.674468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.674472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.674475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.674478:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.674481:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.674488:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.674494:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.674518:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760782.674657:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 27107328<>27111424 00000100:00100000:1.0:1789760782.674842:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760782.674868:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495930496:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.726957:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973935 gen 2 00000008:00010000:1.0:1789760782.727076:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760782.727086:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.727102:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.727132:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760782.727201:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495930496:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.727252:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6618, 6618] 00000080:00200000:0.0:1789760782.727270:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.727277:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.727294:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27111424 i_blocks 53257 00000020:00200000:0.0:1789760782.727297:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.727317:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.727351:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27111424, count: 4096 00000080:00200000:0.0:1789760782.727358:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27111424, bytes: 4096 00000080:00200000:0.0:1789760782.727369:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.727377:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.727379:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27111424, 27115520) 0 0 00000080:00200000:0.0:1789760782.727383:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.727400:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6619, 6619] 00020000:00200000:0.0:1789760782.727409:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27111424 chunk: [27111424, 27115520) 27115520, 4096 00020000:00200000:0.0:1789760782.727414:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.727434:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6619, 6619] 00000008:00010000:0.0:1789760782.727451:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.727457:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.727470:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27111424, 27115520) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.727484:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27111424=0x19db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.727545:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.727548:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27115520 00000020:00010000:0.0:1789760782.727556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.727560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.727564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.727568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.727571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.727579:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.727586:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.727603:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760782.727742:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 27111424<>27115520 00000100:00100000:1.0:1789760782.728364:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.728386:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495930624:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.769549:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973936 gen 2 00000008:00010000:1.0:1789760782.769609:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760782.769619:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.769632:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.769657:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760782.769688:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495930624:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.770030:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6619, 6619] 00000080:00200000:0.0:1789760782.770047:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.770054:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.770071:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27115520 i_blocks 53257 00000020:00200000:0.0:1789760782.770074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.770084:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.770118:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27115520, count: 4096 00000080:00200000:0.0:1789760782.770125:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27115520, bytes: 4096 00000080:00200000:0.0:1789760782.770134:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.770143:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.770145:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27115520, 27119616) 0 0 00000080:00200000:0.0:1789760782.770149:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.770166:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6620, 6620] 00020000:00200000:0.0:1789760782.770206:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27115520 chunk: [27115520, 27119616) 27119616, 4096 00020000:00200000:0.0:1789760782.770211:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.770233:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6620, 6620] 00000008:00010000:0.0:1789760782.770251:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.770258:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.770270:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27115520, 27119616) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.770284:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27115520=0x19dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.770353:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.770356:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27119616 00000020:00010000:0.0:1789760782.770365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.770369:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.770372:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.770376:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.770379:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.770386:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.770393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.770411:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760782.770537:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 27115520<>27119616 00000100:00100000:1.0:1789760782.771295:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760782.771334:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495930752:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760782.796014:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760782.812847:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973937 gen 2 00000008:00010000:1.0:1789760782.812902:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760782.812911:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.812923:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.813968:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760782.814009:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495930752:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.815892:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6620, 6620] 00000080:00200000:0.0:1789760782.815919:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.815925:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.816636:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27119616 i_blocks 53257 00000020:00200000:0.0:1789760782.816644:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.816661:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.816694:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27119616, count: 4096 00000080:00200000:0.0:1789760782.816703:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27119616, bytes: 4096 00000080:00200000:0.0:1789760782.816718:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.816727:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.816729:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27119616, 27123712) 0 0 00000080:00200000:0.0:1789760782.816734:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.816752:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6621, 6621] 00020000:00200000:0.0:1789760782.816761:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27119616 chunk: [27119616, 27123712) 27123712, 4096 00020000:00200000:0.0:1789760782.816767:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.816788:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6621, 6621] 00000008:00010000:0.0:1789760782.816807:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.816813:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.816826:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27119616, 27123712) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.816842:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27119616=0x19dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.816915:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.816919:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27123712 00000020:00010000:0.0:1789760782.816927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.816931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.816935:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.817039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.817045:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.817054:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.817064:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.817095:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760782.817520:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 27119616<>27123712 00000100:00100000:1.0:1789760782.817589:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495930880:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.862688:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973938 gen 2 00000008:00010000:1.0:1789760782.862741:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760782.862750:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.862761:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.862788:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760782.862820:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495930880:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760782.863052:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6621, 6621] 00000080:00200000:2.0:1789760782.863070:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760782.863076:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760782.863093:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27123712 i_blocks 53257 00000020:00200000:2.0:1789760782.863097:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760782.863107:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760782.863143:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27123712, count: 4096 00000080:00200000:2.0:1789760782.863150:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27123712, bytes: 4096 00000080:00200000:2.0:1789760782.863161:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760782.863205:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760782.863207:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27123712, 27127808) 0 0 00000080:00200000:2.0:1789760782.863211:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760782.863229:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6622, 6622] 00020000:00200000:2.0:1789760782.863238:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27123712 chunk: [27123712, 27127808) 27127808, 4096 00020000:00200000:2.0:1789760782.863243:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760782.863266:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6622, 6622] 00000008:00010000:2.0:1789760782.863287:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760782.863292:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760782.863316:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27123712, 27127808) append 0 append lockpos 0 00000080:00200000:2.0:1789760782.863330:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27123712=0x19de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760782.863522:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760782.863525:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27127808 00000020:00010000:2.0:1789760782.863534:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760782.863538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.863541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.863545:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760782.863548:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760782.863557:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760782.863565:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760782.863583:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760782.863772:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 27123712<>27127808 00000100:00100000:3.0:1789760782.863891:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495931008:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760782.905811:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760782.905827:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.905839:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.905858:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760782.905887:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495931008:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760782.906420:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6622, 6622] 00000080:00200000:0.0:1789760782.906438:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.906444:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.906460:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27127808 i_blocks 53257 00000020:00200000:0.0:1789760782.906463:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.906472:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.906508:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27127808, count: 4096 00000080:00200000:0.0:1789760782.906515:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27127808, bytes: 4096 00000080:00200000:0.0:1789760782.906526:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.906536:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.906538:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27127808, 27131904) 0 0 00000080:00200000:0.0:1789760782.906543:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.906561:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6623, 6623] 00020000:00200000:0.0:1789760782.906569:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27127808 chunk: [27127808, 27131904) 27131904, 4096 00020000:00200000:0.0:1789760782.906573:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.906592:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6623, 6623] 00000008:00010000:0.0:1789760782.906608:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.906613:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760782.906892:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27127808, 27131904) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.906909:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27127808=0x19df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.907253:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.907258:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27131904 00000020:00010000:0.0:1789760782.907267:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.907270:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.907274:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.907277:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.907280:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.907289:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.907295:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.907615:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760782.907834:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 27127808<>27131904 00000100:00100000:1.0:1789760782.908222:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495931136:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760782.939942:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973939 gen 2 00000100:00100000:1.0:1789760782.939994:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211836a00 x1876699495931136/t4294973940(4294973940) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760798 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760782.940045:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760782.940052:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.940063:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.940087:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760782.940112:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495931136:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760782.940232:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6623, 6623] 00000080:00200000:2.0:1789760782.940247:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760782.940252:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760782.940265:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27131904 i_blocks 53257 00000020:00200000:2.0:1789760782.940268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760782.940276:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760782.940320:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27131904, count: 4096 00000080:00200000:2.0:1789760782.940328:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27131904, bytes: 4096 00000080:00200000:2.0:1789760782.940339:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760782.940348:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760782.940350:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27131904, 27136000) 0 0 00000080:00200000:2.0:1789760782.940354:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760782.940371:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6624, 6624] 00020000:00200000:2.0:1789760782.940381:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27131904 chunk: [27131904, 27136000) 27136000, 4096 00020000:00200000:2.0:1789760782.940386:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760782.940406:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6624, 6624] 00000008:00010000:2.0:1789760782.940423:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760782.940428:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760782.940439:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27131904, 27136000) append 0 append lockpos 0 00000080:00200000:2.0:1789760782.940454:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27131904=0x19e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760782.940510:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760782.940514:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27136000 00000020:00010000:2.0:1789760782.940523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760782.940527:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.940529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.940532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760782.940535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760782.940542:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760782.940549:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760782.940566:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760782.940816:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 27131904<>27136000 00000100:00100000:3.0:1789760782.940894:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495931264:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760782.988774:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973941 gen 2 00000008:00010000:2.0:1789760782.988840:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760782.988848:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.988858:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.988885:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760782.988916:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495931264:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760782.991396:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6624, 6624] 00000080:00200000:1.0:1789760782.991425:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760782.991432:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760782.991450:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27136000 i_blocks 53257 00000020:00200000:1.0:1789760782.991456:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760782.991471:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760782.991507:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27136000, count: 4096 00000080:00200000:1.0:1789760782.991515:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27136000, bytes: 4096 00000080:00200000:1.0:1789760782.991528:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760782.991537:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760782.991540:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27136000, 27140096) 0 0 00000080:00200000:1.0:1789760782.991544:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760782.991569:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6625, 6625] 00020000:00200000:1.0:1789760782.991583:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27136000 chunk: [27136000, 27140096) 27140096, 4096 00020000:00200000:1.0:1789760782.991588:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760782.991611:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6625, 6625] 00000008:00010000:1.0:1789760782.991631:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760782.991638:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760782.991650:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27136000, 27140096) append 0 append lockpos 0 00000080:00200000:1.0:1789760782.991665:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27136000=0x19e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760782.991736:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760782.991739:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27140096 00000020:00010000:1.0:1789760782.991749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760782.991754:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760782.991757:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760782.991760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760782.991763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760782.991771:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760782.991778:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760782.991802:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760782.991948:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27136000<>27140096 00000100:00100000:1.0:1789760782.992020:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.992049:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495931392:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760783.029049:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760783.029064:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.029076:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.029097:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760783.029127:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495931392:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.029239:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6625, 6625] 00000080:00200000:0.0:1789760783.029256:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.029263:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.029280:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27140096 i_blocks 53257 00000020:00200000:0.0:1789760783.029285:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.029294:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.029338:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27140096, count: 4096 00000080:00200000:0.0:1789760783.029347:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27140096, bytes: 4096 00000080:00200000:0.0:1789760783.029359:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.029369:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.029371:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27140096, 27144192) 0 0 00000080:00200000:0.0:1789760783.029376:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.029394:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6626, 6626] 00020000:00200000:0.0:1789760783.029402:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27140096 chunk: [27140096, 27144192) 27144192, 4096 00020000:00200000:0.0:1789760783.029407:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.029426:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6626, 6626] 00000008:00010000:0.0:1789760783.029444:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.029450:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.029462:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27140096, 27144192) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.029477:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27140096=0x19e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.029536:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.029539:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27144192 00000020:00010000:0.0:1789760783.029547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.029551:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.029554:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.029557:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.029560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.029568:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.029575:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.029592:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760783.031019:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 27140096<>27144192 00000100:00100000:0.0:1789760783.031084:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760783.031108:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495931520:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.118214:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973943 gen 2 00000008:00010000:1.0:1789760783.118284:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760783.118293:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.118316:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.118343:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760783.118378:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495931520:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.118413:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6626, 6626] 00000080:00200000:0.0:1789760783.118436:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.118444:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.118461:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27144192 i_blocks 53257 00000020:00200000:0.0:1789760783.118465:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.118475:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.118507:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27144192, count: 4096 00000080:00200000:0.0:1789760783.118514:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27144192, bytes: 4096 00000080:00200000:0.0:1789760783.118528:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.118538:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.118540:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27144192, 27148288) 0 0 00000080:00200000:0.0:1789760783.118545:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.118561:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6627, 6627] 00020000:00200000:0.0:1789760783.118570:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27144192 chunk: [27144192, 27148288) 27148288, 4096 00020000:00200000:0.0:1789760783.118575:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.118594:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6627, 6627] 00000008:00010000:0.0:1789760783.118612:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.118618:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.118630:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27144192, 27148288) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.118644:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27144192=0x19e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.118703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.118706:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27148288 00000020:00010000:0.0:1789760783.118716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.118719:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.118722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.118725:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.118728:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.118736:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.118744:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.118759:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760783.118898:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 27144192<>27148288 00000100:00100000:1.0:1789760783.119401:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760783.119422:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495931648:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.178251:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973944 gen 2 00000008:00010000:1.0:1789760783.178312:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760783.178321:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.178333:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.178354:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760783.178384:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495931648:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.178487:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6627, 6627] 00000080:00200000:0.0:1789760783.178503:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.178510:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.178526:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27148288 i_blocks 53257 00000020:00200000:0.0:1789760783.178530:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.178544:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.178577:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27148288, count: 4096 00000080:00200000:0.0:1789760783.178584:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27148288, bytes: 4096 00000080:00200000:0.0:1789760783.178593:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.178603:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.178605:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27148288, 27152384) 0 0 00000080:00200000:0.0:1789760783.178609:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.178626:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6628, 6628] 00020000:00200000:0.0:1789760783.178634:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27148288 chunk: [27148288, 27152384) 27152384, 4096 00020000:00200000:0.0:1789760783.178639:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.178658:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6628, 6628] 00000008:00010000:0.0:1789760783.178677:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.178683:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.178694:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27148288, 27152384) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.178708:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27148288=0x19e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.178759:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.178762:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27152384 00000020:00010000:0.0:1789760783.178770:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.178774:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.178777:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.178780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.178783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.178792:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.178799:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.178815:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760783.179785:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 27148288<>27152384 00000100:00100000:1.0:1789760783.179846:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495931776:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.241249:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973945 gen 2 00000008:00010000:1.0:1789760783.241311:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760783.241320:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.241333:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.241398:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760783.241429:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495931776:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.242843:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6628, 6628] 00000080:00200000:0.0:1789760783.242879:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.242887:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.242907:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27152384 i_blocks 53257 00000020:00200000:0.0:1789760783.242912:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.242925:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.242962:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27152384, count: 4096 00000080:00200000:0.0:1789760783.242970:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27152384, bytes: 4096 00000080:00200000:0.0:1789760783.242984:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.242994:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.242996:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27152384, 27156480) 0 0 00000080:00200000:0.0:1789760783.243000:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.243018:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6629, 6629] 00020000:00200000:0.0:1789760783.243026:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27152384 chunk: [27152384, 27156480) 27156480, 4096 00020000:00200000:0.0:1789760783.243031:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.243053:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6629, 6629] 00000008:00010000:0.0:1789760783.243072:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.243078:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.243090:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27152384, 27156480) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.243106:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27152384=0x19e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.243209:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.243212:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27156480 00000020:00010000:0.0:1789760783.243221:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.243225:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.243228:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.243231:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.243234:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.243245:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.243252:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.243281:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760783.243436:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 27152384<>27156480 00000100:00100000:1.0:1789760783.243681:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495931904:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.290074:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973946 gen 2 00000008:00010000:1.0:1789760783.290127:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760783.290135:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.290146:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.290255:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760783.290282:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495931904:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.290446:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6629, 6629] 00000080:00200000:0.0:1789760783.290461:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.290466:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.290482:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27156480 i_blocks 53257 00000020:00200000:0.0:1789760783.290485:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.290494:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.290695:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27156480, count: 4096 00000080:00200000:0.0:1789760783.290704:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27156480, bytes: 4096 00000080:00200000:0.0:1789760783.290717:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.290726:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.290728:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27156480, 27160576) 0 0 00000080:00200000:0.0:1789760783.290733:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.290752:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6630, 6630] 00020000:00200000:0.0:1789760783.290761:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27156480 chunk: [27156480, 27160576) 27160576, 4096 00020000:00200000:0.0:1789760783.290765:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.290786:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6630, 6630] 00000008:00010000:0.0:1789760783.290805:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.290811:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.290823:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27156480, 27160576) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.290836:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27156480=0x19e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.290899:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.290902:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27160576 00000020:00010000:0.0:1789760783.290910:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.290914:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.290917:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.290920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.290923:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.290932:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.290939:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.290957:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760783.291085:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 27156480<>27160576 00000100:00100000:1.0:1789760783.291245:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495932032:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.326503:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973947 gen 2 00000008:00010000:1.0:1789760783.326557:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760783.326564:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.326920:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.326994:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760783.327026:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495932032:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.329983:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6630, 6630] 00000080:00200000:0.0:1789760783.330020:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.330028:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.330087:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27160576 i_blocks 53257 00000020:00200000:0.0:1789760783.330094:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.330112:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.330145:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27160576, count: 4096 00000080:00200000:0.0:1789760783.330155:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27160576, bytes: 4096 00000080:00200000:0.0:1789760783.330275:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.330286:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.330288:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27160576, 27164672) 0 0 00000080:00200000:0.0:1789760783.331444:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.331474:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6631, 6631] 00020000:00200000:0.0:1789760783.331485:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27160576 chunk: [27160576, 27164672) 27164672, 4096 00020000:00200000:0.0:1789760783.331490:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.331518:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6631, 6631] 00000008:00010000:0.0:1789760783.331545:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.331553:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.331567:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27160576, 27164672) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.331585:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27160576=0x19e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.331662:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.331665:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27164672 00000020:00010000:0.0:1789760783.331674:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.331678:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.331681:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.331685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.331688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.331699:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.331707:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.331737:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760783.331882:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 27160576<>27164672 00000100:00100000:1.0:1789760783.331985:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932160:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.373933:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973948 gen 2 00000008:00010000:1.0:1789760783.373997:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760783.374005:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.374017:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.374039:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000080:00200000:0.0:1789760783.374142:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6631, 6631] 00000080:00200000:0.0:1789760783.374161:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.374205:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.374219:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27164672 i_blocks 53257 00000020:00200000:0.0:1789760783.374223:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.374232:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.374262:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27164672, count: 4096 00000080:00200000:0.0:1789760783.374269:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27164672, bytes: 4096 00000080:00200000:0.0:1789760783.374281:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.374289:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.374291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27164672, 27168768) 0 0 00000080:00200000:0.0:1789760783.374304:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.374320:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6632, 6632] 00020000:00200000:0.0:1789760783.374328:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27164672 chunk: [27164672, 27168768) 27168768, 4096 00020000:00200000:0.0:1789760783.374333:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.374352:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6632, 6632] 00000008:00010000:0.0:1789760783.374369:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.374374:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.374384:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27164672, 27168768) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.374396:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27164672=0x19e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.374447:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.374450:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27168768 00000020:00010000:0.0:1789760783.374457:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.374461:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.374465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.374469:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.374472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.374480:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.374488:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760783.374628:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932160:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760783.374673:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:1.0:1789760783.374915:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27164672<>27168768 00000100:00100000:1.0:1789760783.374957:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760783.375036:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932288:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760783.418622:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760783.418638:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.418648:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.418778:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760783.418809:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932288:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.419086:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6632, 6632] 00000080:00200000:0.0:1789760783.419103:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.419110:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.419127:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27168768 i_blocks 53257 00000020:00200000:0.0:1789760783.419130:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.419140:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.419206:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27168768, count: 4096 00000080:00200000:0.0:1789760783.419214:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27168768, bytes: 4096 00000080:00200000:0.0:1789760783.419223:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.419232:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.419235:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27168768, 27172864) 0 0 00000080:00200000:0.0:1789760783.419240:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.419453:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6633, 6633] 00020000:00200000:0.0:1789760783.419462:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27168768 chunk: [27168768, 27172864) 27172864, 4096 00020000:00200000:0.0:1789760783.419468:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.419906:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6633, 6633] 00000008:00010000:0.0:1789760783.419931:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.419937:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.419950:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27168768, 27172864) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.419967:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27168768=0x19e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.422413:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.422423:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27172864 00000020:00010000:0.0:1789760783.422439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.422445:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.422450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.422455:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.422457:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.422470:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.422482:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.422519:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760783.422773:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 27168768<>27172864 00000100:00100000:1.0:1789760783.422853:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932416:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.464756:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973950 gen 2 00000008:00010000:1.0:1789760783.464816:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760783.464825:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.464835:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.464861:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760783.464890:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932416:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.465395:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6633, 6633] 00000080:00200000:0.0:1789760783.465414:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.465421:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.465436:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27172864 i_blocks 53257 00000020:00200000:0.0:1789760783.465440:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.465449:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.465484:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27172864, count: 4096 00000080:00200000:0.0:1789760783.465492:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27172864, bytes: 4096 00000080:00200000:0.0:1789760783.465501:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.465509:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.465512:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27172864, 27176960) 0 0 00000080:00200000:0.0:1789760783.465517:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.465534:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6634, 6634] 00020000:00200000:0.0:1789760783.465542:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27172864 chunk: [27172864, 27176960) 27176960, 4096 00020000:00200000:0.0:1789760783.465547:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.465567:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6634, 6634] 00000008:00010000:0.0:1789760783.465584:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.465590:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.465602:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27172864, 27176960) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.465616:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27172864=0x19ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.465675:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.465678:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27176960 00000020:00010000:0.0:1789760783.465687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.465690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.465693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.465697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.465700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.465707:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.465714:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.465731:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760783.465855:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 27172864<>27176960 00000100:00100000:1.0:1789760783.466431:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495932544:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.505489:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973951 gen 2 00000008:00010000:1.0:1789760783.505543:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760783.505551:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.505563:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.505582:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760783.505611:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495932544:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760783.505784:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6634, 6634] 00000080:00200000:0.0:1789760783.505802:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.505808:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.505824:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27176960 i_blocks 53257 00000020:00200000:0.0:1789760783.505828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.505838:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.505873:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27176960, count: 4096 00000080:00200000:0.0:1789760783.505880:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27176960, bytes: 4096 00000080:00200000:0.0:1789760783.505889:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.505899:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.505901:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27176960, 27181056) 0 0 00000080:00200000:0.0:1789760783.505906:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.505922:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6635, 6635] 00020000:00200000:0.0:1789760783.505931:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27176960 chunk: [27176960, 27181056) 27181056, 4096 00020000:00200000:0.0:1789760783.505936:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.505957:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6635, 6635] 00000008:00010000:0.0:1789760783.505976:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.505982:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760783.505994:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27176960, 27181056) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.506008:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27176960=0x19eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.506067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.506071:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27181056 00000020:00010000:0.0:1789760783.506080:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.506084:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.506087:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.506090:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.506093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.506101:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.506108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.506126:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760783.506510:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 27176960<>27181056 00000100:00100000:1.0:1789760783.506585:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932672:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.549536:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973952 gen 2 00000008:00010000:1.0:1789760783.549589:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760783.549597:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.549608:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.549641:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760783.549672:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932672:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.550355:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6635, 6635] 00000080:00200000:2.0:1789760783.550373:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.550380:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.550395:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27181056 i_blocks 53257 00000020:00200000:2.0:1789760783.550398:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.550408:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.550441:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27181056, count: 4096 00000080:00200000:2.0:1789760783.550448:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27181056, bytes: 4096 00000080:00200000:2.0:1789760783.550462:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.550471:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.550474:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27181056, 27185152) 0 0 00000080:00200000:2.0:1789760783.550479:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.550496:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6636, 6636] 00020000:00200000:2.0:1789760783.550504:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27181056 chunk: [27181056, 27185152) 27185152, 4096 00020000:00200000:2.0:1789760783.550510:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.550534:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6636, 6636] 00000008:00010000:2.0:1789760783.550554:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.550560:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.550572:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27181056, 27185152) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.550587:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27181056=0x19ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.550646:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.550650:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27185152 00000020:00010000:2.0:1789760783.550658:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.550662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.550665:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.550668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.550672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.550679:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.550686:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.550703:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760783.550827:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 27181056<>27185152 00000100:00100000:2.0:1789760783.550872:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760783.550893:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495932800:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760783.600483:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973953 gen 2 00000008:00010000:2.0:1789760783.600532:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:2.0:1789760783.600540:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.600552:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.600581:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:2.0:1789760783.600610:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495932800:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760783.600893:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6636, 6636] 00000080:00200000:1.0:1789760783.600910:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760783.600916:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760783.600932:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27185152 i_blocks 53257 00000020:00200000:1.0:1789760783.600936:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760783.600948:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760783.600993:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27185152, count: 4096 00000080:00200000:1.0:1789760783.601001:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27185152, bytes: 4096 00000080:00200000:1.0:1789760783.601012:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760783.601021:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760783.601024:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27185152, 27189248) 0 0 00000080:00200000:1.0:1789760783.601028:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760783.601047:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6637, 6637] 00020000:00200000:1.0:1789760783.601056:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27185152 chunk: [27185152, 27189248) 27189248, 4096 00020000:00200000:1.0:1789760783.601061:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760783.601082:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6637, 6637] 00000008:00010000:1.0:1789760783.601102:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760783.601108:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760783.601119:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27185152, 27189248) append 0 append lockpos 0 00000080:00200000:1.0:1789760783.601133:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27185152=0x19ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760783.601221:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760783.601225:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27189248 00000020:00010000:1.0:1789760783.601233:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760783.601237:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.601240:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.601244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760783.601247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760783.601254:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760783.601261:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760783.601299:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760783.601424:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27185152<>27189248 00000100:00100000:1.0:1789760783.601469:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760783.601488:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932928:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760783.632904:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973954 gen 2 00000008:00010000:1.0:1789760783.632957:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760783.632977:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.632988:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.633014:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760783.633045:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495932928:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.633220:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6637, 6637] 00000080:00200000:2.0:1789760783.633239:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.633245:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.633261:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27189248 i_blocks 53257 00000020:00200000:2.0:1789760783.633265:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.633275:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.633318:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27189248, count: 4096 00000080:00200000:2.0:1789760783.633325:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27189248, bytes: 4096 00000080:00200000:2.0:1789760783.633334:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.633342:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.633344:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27189248, 27193344) 0 0 00000080:00200000:2.0:1789760783.633348:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.633364:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6638, 6638] 00020000:00200000:2.0:1789760783.633372:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27189248 chunk: [27189248, 27193344) 27193344, 4096 00020000:00200000:2.0:1789760783.633376:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.633394:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6638, 6638] 00000008:00010000:2.0:1789760783.633412:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.633417:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.633428:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27189248, 27193344) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.633442:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27189248=0x19ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.633503:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.633506:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27193344 00000020:00010000:2.0:1789760783.633514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.633518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.633521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.633525:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.633528:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.633535:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.633542:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.633559:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760783.633747:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 27189248<>27193344 00000100:00100000:3.0:1789760783.633854:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760783.633872:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495933056:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760783.674491:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760783.674506:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.674517:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.674588:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760783.674631:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495933056:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.675574:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6638, 6638] 00000080:00200000:2.0:1789760783.675599:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.675615:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.675632:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27193344 i_blocks 53257 00000020:00200000:2.0:1789760783.675638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.675649:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.675690:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27193344, count: 4096 00000080:00200000:2.0:1789760783.675698:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27193344, bytes: 4096 00000080:00200000:2.0:1789760783.675711:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.675720:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.675722:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27193344, 27197440) 0 0 00000080:00200000:2.0:1789760783.675727:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.675746:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6639, 6639] 00020000:00200000:2.0:1789760783.675755:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27193344 chunk: [27193344, 27197440) 27197440, 4096 00020000:00200000:2.0:1789760783.675759:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.675793:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6639, 6639] 00000008:00010000:2.0:1789760783.675812:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.675818:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.675830:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27193344, 27197440) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.677142:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27193344=0x19ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.678281:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.678356:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27197440 00000020:00010000:2.0:1789760783.678371:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.678383:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.678387:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.678391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.678394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.678406:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.678416:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.678448:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760783.678725:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 27193344<>27197440 00000100:00100000:3.0:1789760783.678827:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495933184:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760783.711985:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973955 gen 2 00000100:00100000:3.0:1789760783.712024:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495933184/t4294973956(4294973956) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760799 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760783.712073:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760783.712081:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.712092:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.712149:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000080:00200000:2.0:1789760783.712154:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6639, 6639] 00000100:00100000:3.0:1789760783.712208:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495933184:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.712211:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.712218:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.712234:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27197440 i_blocks 53257 00000020:00200000:2.0:1789760783.712239:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.712248:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.712284:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27197440, count: 4096 00000080:00200000:2.0:1789760783.712304:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27197440, bytes: 4096 00000080:00200000:2.0:1789760783.712315:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.712325:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.712327:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27197440, 27201536) 0 0 00000080:00200000:2.0:1789760783.712332:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.712349:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6640, 6640] 00020000:00200000:2.0:1789760783.712356:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27197440 chunk: [27197440, 27201536) 27201536, 4096 00020000:00200000:2.0:1789760783.712359:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.712377:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6640, 6640] 00000008:00010000:2.0:1789760783.712391:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.712396:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.712406:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27197440, 27201536) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.712417:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27197440=0x19f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.712590:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.712594:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27201536 00000020:00010000:2.0:1789760783.712604:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.712608:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.712611:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.712646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.712649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.712659:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.712667:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.712685:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760783.712828:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 27197440<>27201536 00000100:00100000:3.0:1789760783.712945:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495933312:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760783.751741:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973957 gen 2 00000008:00010000:3.0:1789760783.751804:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760783.751813:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.751823:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.751852:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760783.751883:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495933312:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.752199:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6640, 6640] 00000080:00200000:2.0:1789760783.752215:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.752221:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.752236:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27201536 i_blocks 53257 00000020:00200000:2.0:1789760783.752241:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.752249:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.752301:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27201536, count: 4096 00000080:00200000:2.0:1789760783.752307:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27201536, bytes: 4096 00000080:00200000:2.0:1789760783.752317:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.752325:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.752328:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27201536, 27205632) 0 0 00000080:00200000:2.0:1789760783.752332:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.752347:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6641, 6641] 00020000:00200000:2.0:1789760783.752355:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27201536 chunk: [27201536, 27205632) 27205632, 4096 00020000:00200000:2.0:1789760783.752359:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.752376:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6641, 6641] 00000008:00010000:2.0:1789760783.752392:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.752398:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.752409:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27201536, 27205632) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.752423:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27201536=0x19f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.752480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.752483:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27205632 00000020:00010000:2.0:1789760783.752491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.752495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.752498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.752502:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.752504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.752511:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.752517:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.752534:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760783.752664:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 27201536<>27205632 00000100:00100000:3.0:1789760783.753242:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760783.753275:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495933440:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760783.793025:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973958 gen 2 00000008:00010000:3.0:1789760783.793078:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760783.793087:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.793098:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.793123:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760783.793154:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495933440:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.796651:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6641, 6641] 00000080:00200000:2.0:1789760783.796680:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.796687:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.796706:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27205632 i_blocks 53257 00000020:00200000:2.0:1789760783.796712:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.796729:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.796762:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27205632, count: 4096 00000080:00200000:2.0:1789760783.796771:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27205632, bytes: 4096 00000080:00200000:2.0:1789760783.796787:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.796796:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.796799:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27205632, 27209728) 0 0 00000080:00200000:2.0:1789760783.796803:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.796819:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6642, 6642] 00020000:00200000:2.0:1789760783.796828:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27205632 chunk: [27205632, 27209728) 27209728, 4096 00020000:00200000:2.0:1789760783.796833:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.796855:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6642, 6642] 00000008:00010000:2.0:1789760783.796874:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.796881:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.796893:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27205632, 27209728) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.796907:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27205632=0x19f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.796975:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.796978:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27209728 00000020:00010000:2.0:1789760783.796986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.796990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.796993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.796997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.797000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.797009:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.797016:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.797038:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760783.797223:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 27205632<>27209728 00000100:00100000:3.0:1789760783.799601:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495933568:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760783.819434:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760783.834423:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973959 gen 2 00000008:00010000:3.0:1789760783.834479:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760783.834487:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.834496:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.834513:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760783.834542:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495933568:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.834897:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6642, 6642] 00000080:00200000:2.0:1789760783.834914:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.834920:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.834935:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27209728 i_blocks 53257 00000020:00200000:2.0:1789760783.834939:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.834949:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.834981:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27209728, count: 4096 00000080:00200000:2.0:1789760783.834988:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27209728, bytes: 4096 00000080:00200000:2.0:1789760783.834997:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.835006:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.835008:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27209728, 27213824) 0 0 00000080:00200000:2.0:1789760783.835013:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.835029:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6643, 6643] 00020000:00200000:2.0:1789760783.835037:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27209728 chunk: [27209728, 27213824) 27213824, 4096 00020000:00200000:2.0:1789760783.835042:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.835063:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6643, 6643] 00000008:00010000:2.0:1789760783.835082:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.835088:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.835099:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27209728, 27213824) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.835113:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27209728=0x19f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.835706:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.835711:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27213824 00000020:00010000:2.0:1789760783.835722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.835726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.835730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.835734:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.835737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.835745:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.835752:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.835771:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088130c0 00000008:00100000:2.0:1789760783.835895:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 27209728<>27213824 00000100:00100000:3.0:1789760783.837827:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495933696:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760783.885906:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088130c0 00000020:00200000:3.0:1789760783.885924:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.885936:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.885963:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088130c0 00000100:00100000:3.0:1789760783.885992:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495933696:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.886153:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6643, 6643] 00000080:00200000:2.0:1789760783.886203:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.886210:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.886225:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27213824 i_blocks 53257 00000020:00200000:2.0:1789760783.886229:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.886239:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.886282:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27213824, count: 4096 00000080:00200000:2.0:1789760783.886298:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27213824, bytes: 4096 00000080:00200000:2.0:1789760783.886308:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.886318:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.886320:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27213824, 27217920) 0 0 00000080:00200000:2.0:1789760783.886325:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.886353:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6644, 6644] 00020000:00200000:2.0:1789760783.886362:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27213824 chunk: [27213824, 27217920) 27217920, 4096 00020000:00200000:2.0:1789760783.886367:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.886387:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6644, 6644] 00000008:00010000:2.0:1789760783.886405:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.886411:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.886422:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27213824, 27217920) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.886437:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27213824=0x19f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.886492:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.886495:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27217920 00000020:00010000:2.0:1789760783.886504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.886508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.886512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.886515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.886518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.886526:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.886532:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.886548:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760783.886837:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 27213824<>27217920 00000100:00100000:3.0:1789760783.887002:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495933824:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760783.921092:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973961 gen 2 00000008:00010000:3.0:1789760783.921143:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760783.921150:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.921157:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.921213:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760783.921242:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495933824:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.921437:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6644, 6644] 00000080:00200000:2.0:1789760783.921454:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.921461:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.921476:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27217920 i_blocks 53257 00000020:00200000:2.0:1789760783.921480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.921489:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.921525:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27217920, count: 4096 00000080:00200000:2.0:1789760783.921533:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27217920, bytes: 4096 00000080:00200000:2.0:1789760783.921542:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.921550:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.921553:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27217920, 27222016) 0 0 00000080:00200000:2.0:1789760783.921557:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.921575:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6645, 6645] 00020000:00200000:2.0:1789760783.921585:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27217920 chunk: [27217920, 27222016) 27222016, 4096 00020000:00200000:2.0:1789760783.921590:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.921767:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6645, 6645] 00000008:00010000:2.0:1789760783.921788:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.921795:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.921807:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27217920, 27222016) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.921822:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27217920=0x19f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.921939:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.921942:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27222016 00000020:00010000:2.0:1789760783.921951:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.921954:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.921957:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.921961:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.921963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.921971:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.921978:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.921996:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760783.922123:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 27217920<>27222016 00000100:00100000:3.0:1789760783.922199:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495933952:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760783.955914:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760783.955929:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.955940:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.956002:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760783.956032:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495933952:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760783.956098:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6645, 6645] 00000080:00200000:2.0:1789760783.956114:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.956120:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.956135:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27222016 i_blocks 53257 00000020:00200000:2.0:1789760783.956139:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.956148:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.956395:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27222016, count: 4096 00000080:00200000:2.0:1789760783.956404:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27222016, bytes: 4096 00000080:00200000:2.0:1789760783.956414:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.956422:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.956424:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27222016, 27226112) 0 0 00000080:00200000:2.0:1789760783.956429:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.956445:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6646, 6646] 00020000:00200000:2.0:1789760783.956454:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27222016 chunk: [27222016, 27226112) 27226112, 4096 00020000:00200000:2.0:1789760783.956458:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.956478:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6646, 6646] 00000008:00010000:2.0:1789760783.956682:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.956687:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760783.956698:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27222016, 27226112) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.956711:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27222016=0x19f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.956764:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.956766:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27226112 00000020:00010000:2.0:1789760783.956774:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.956777:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.956780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.956783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.956786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.956794:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.956800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.956817:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760783.960466:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 27222016<>27226112 00000100:00100000:3.0:1789760783.960574:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760783.960923:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495934080:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760783.981548:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973963 gen 2 00000008:00010000:3.0:1789760783.981614:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760783.981623:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.981633:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.981662:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760783.981702:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495934080:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760783.981892:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6646, 6646] 00000080:00200000:1.0:1789760783.981908:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760783.981914:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760783.981931:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27226112 i_blocks 53257 00000020:00200000:1.0:1789760783.981934:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760783.981944:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760783.981979:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27226112, count: 4096 00000080:00200000:1.0:1789760783.981986:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27226112, bytes: 4096 00000080:00200000:1.0:1789760783.981996:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760783.982004:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760783.982006:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27226112, 27230208) 0 0 00000080:00200000:1.0:1789760783.982011:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760783.982026:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6647, 6647] 00020000:00200000:1.0:1789760783.982035:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27226112 chunk: [27226112, 27230208) 27230208, 4096 00020000:00200000:1.0:1789760783.982040:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760783.982059:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6647, 6647] 00000008:00010000:1.0:1789760783.982077:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760783.982083:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760783.982094:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27226112, 27230208) append 0 append lockpos 0 00000080:00200000:1.0:1789760783.982108:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27226112=0x19f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760783.982163:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760783.982166:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27230208 00000020:00010000:1.0:1789760783.982219:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760783.982224:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.982227:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.982231:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760783.982234:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760783.982242:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760783.982249:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760783.982264:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760783.982533:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27226112<>27230208 00000100:00100000:0.0:1789760783.982594:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760783.983550:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934208:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.007011:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973964 gen 2 00000008:00010000:1.0:1789760784.007071:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760784.007080:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.007091:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.007109:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760784.007137:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934208:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.007249:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6647, 6647] 00000080:00200000:0.0:1789760784.007264:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.007269:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.007283:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27230208 i_blocks 53257 00000020:00200000:0.0:1789760784.007287:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.007305:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.007615:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27230208, count: 4096 00000080:00200000:0.0:1789760784.007623:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27230208, bytes: 4096 00000080:00200000:0.0:1789760784.007635:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.007644:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.007646:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27230208, 27234304) 0 0 00000080:00200000:0.0:1789760784.007650:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.007666:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6648, 6648] 00020000:00200000:0.0:1789760784.007674:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27230208 chunk: [27230208, 27234304) 27234304, 4096 00020000:00200000:0.0:1789760784.007678:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.007697:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6648, 6648] 00000008:00010000:0.0:1789760784.007715:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.007721:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.007733:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27230208, 27234304) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.007749:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27230208=0x19f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.007809:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.007812:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27234304 00000020:00010000:0.0:1789760784.007821:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.007824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.007827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.007831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.007834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.007841:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.007858:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.007877:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760784.008001:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 27230208<>27234304 00000100:00100000:0.0:1789760784.008047:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760784.008063:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495934336:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.039951:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973965 gen 2 00000008:00010000:1.0:1789760784.040003:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760784.040011:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.040023:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.040044:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000080:00200000:0.0:1789760784.040070:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6648, 6648] 00000100:00100000:1.0:1789760784.040075:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495934336:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.040088:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.040094:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.040109:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27234304 i_blocks 53257 00000020:00200000:0.0:1789760784.040112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.040120:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.040154:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27234304, count: 4096 00000080:00200000:0.0:1789760784.040161:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27234304, bytes: 4096 00000080:00200000:0.0:1789760784.040210:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.040219:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.040221:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27234304, 27238400) 0 0 00000080:00200000:0.0:1789760784.040226:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.040475:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6649, 6649] 00020000:00200000:0.0:1789760784.040485:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27234304 chunk: [27234304, 27238400) 27238400, 4096 00020000:00200000:0.0:1789760784.040490:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.040511:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6649, 6649] 00000008:00010000:0.0:1789760784.040529:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.040535:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.040547:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27234304, 27238400) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.040561:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27234304=0x19f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.040609:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.040612:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27238400 00000020:00010000:0.0:1789760784.040620:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.040624:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.040627:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.040631:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.040633:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.040642:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.040650:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.040662:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760784.040780:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 27234304<>27238400 00000100:00100000:1.0:1789760784.040896:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760784.040913:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934464:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.077034:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973966 gen 2 00000008:00010000:1.0:1789760784.077086:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760784.077094:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.077105:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.077124:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760784.077154:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934464:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.077454:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6649, 6649] 00000080:00200000:0.0:1789760784.077475:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.077481:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.077494:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27238400 i_blocks 53257 00000020:00200000:0.0:1789760784.077498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.077508:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.077547:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27238400, count: 4096 00000080:00200000:0.0:1789760784.077554:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27238400, bytes: 4096 00000080:00200000:0.0:1789760784.077563:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.077571:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.077574:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27238400, 27242496) 0 0 00000080:00200000:0.0:1789760784.077578:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.077592:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6650, 6650] 00020000:00200000:0.0:1789760784.077599:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27238400 chunk: [27238400, 27242496) 27242496, 4096 00020000:00200000:0.0:1789760784.077603:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.077628:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6650, 6650] 00000008:00010000:0.0:1789760784.077792:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.077798:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.077809:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27238400, 27242496) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.077820:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27238400=0x19fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.077868:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.077871:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27242496 00000020:00010000:0.0:1789760784.077878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.077882:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.077886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.077889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.077891:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.077898:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.077904:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.077923:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760784.078036:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 27238400<>27242496 00000100:00100000:1.0:1789760784.078163:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934592:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.110075:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973967 gen 2 00000008:00010000:1.0:1789760784.110112:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760784.110128:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.110138:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.110157:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000080:00200000:0.0:1789760784.110319:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6650, 6650] 00000080:00200000:0.0:1789760784.110333:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.110338:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.110350:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27242496 i_blocks 53257 00000020:00200000:0.0:1789760784.110353:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.110361:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.110385:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27242496, count: 4096 00000080:00200000:0.0:1789760784.110392:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27242496, bytes: 4096 00000080:00200000:0.0:1789760784.110401:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.110408:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.110410:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27242496, 27246592) 0 0 00000080:00200000:0.0:1789760784.110414:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.110426:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6651, 6651] 00020000:00200000:0.0:1789760784.110433:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27242496 chunk: [27242496, 27246592) 27246592, 4096 00020000:00200000:0.0:1789760784.110436:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.110452:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6651, 6651] 00000008:00010000:0.0:1789760784.110466:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.110470:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.110478:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27242496, 27246592) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.110488:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27242496=0x19fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.110525:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.110528:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27246592 00000020:00010000:0.0:1789760784.110535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.110539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.110541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.110544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.110559:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.110565:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.110630:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760784.110679:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934592:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760784.111606:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:1.0:1789760784.111709:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27242496<>27246592 00000100:00100000:1.0:1789760784.111741:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760784.111756:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934720:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.164049:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973968 gen 2 00000008:00010000:1.0:1789760784.164092:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760784.164100:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.164108:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.164165:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760784.164215:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934720:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.164403:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6651, 6651] 00000080:00200000:0.0:1789760784.164416:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.164420:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.164431:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27246592 i_blocks 53257 00000020:00200000:0.0:1789760784.164434:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.164442:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.164468:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27246592, count: 4096 00000080:00200000:0.0:1789760784.164474:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27246592, bytes: 4096 00000080:00200000:0.0:1789760784.164481:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.164488:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.164490:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27246592, 27250688) 0 0 00000080:00200000:0.0:1789760784.164494:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.164507:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6652, 6652] 00020000:00200000:0.0:1789760784.164513:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27246592 chunk: [27246592, 27250688) 27250688, 4096 00020000:00200000:0.0:1789760784.164517:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.164533:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6652, 6652] 00000008:00010000:0.0:1789760784.164546:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.164551:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.164559:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27246592, 27250688) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.164570:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27246592=0x19fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.164615:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.164618:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27250688 00000020:00010000:0.0:1789760784.164625:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.164628:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.164630:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.164634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.164636:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.164642:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.164647:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.164660:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760784.164750:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 27246592<>27250688 00000100:00100000:1.0:1789760784.164811:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934848:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.215668:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973969 gen 2 00000008:00010000:1.0:1789760784.215705:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760784.215711:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.215721:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.215845:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760784.215870:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495934848:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.218145:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6652, 6652] 00000080:00200000:0.0:1789760784.218165:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.218183:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.218196:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27250688 i_blocks 53257 00000020:00200000:0.0:1789760784.218201:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.218213:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.218236:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27250688, count: 4096 00000080:00200000:0.0:1789760784.218242:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27250688, bytes: 4096 00000080:00200000:0.0:1789760784.218252:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.218260:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.218262:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27250688, 27254784) 0 0 00000080:00200000:0.0:1789760784.218266:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.218278:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6653, 6653] 00020000:00200000:0.0:1789760784.218285:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27250688 chunk: [27250688, 27254784) 27254784, 4096 00020000:00200000:0.0:1789760784.218296:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.218315:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6653, 6653] 00000008:00010000:0.0:1789760784.218330:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.218335:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.218344:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27250688, 27254784) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.218355:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27250688=0x19fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.218414:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.218417:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27254784 00000020:00010000:0.0:1789760784.218424:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.218427:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.218431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.218435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.218437:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.218444:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.218449:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.218473:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760784.218580:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 27250688<>27254784 00000100:00100000:1.0:1789760784.219076:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495934976:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.271555:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495935232:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760784.271664:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1789760784.271672:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495935104:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760784.271700:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2357:1876699495935360:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760784.274122:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495935104:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760784.274143:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2357:1876699495935360:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760784.274153:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495935232:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760784.292472:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973970 gen 2 00000008:00010000:1.0:1789760784.292510:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760784.292516:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.292537:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.292555:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760784.292579:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495934976:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.292598:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6653, 6653] 00000080:00200000:0.0:1789760784.292612:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.292628:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.292640:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27254784 i_blocks 53257 00000020:00200000:0.0:1789760784.292644:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.292653:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.292675:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27254784, count: 4096 00000080:00200000:0.0:1789760784.292682:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27254784, bytes: 4096 00000080:00200000:0.0:1789760784.292689:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.292695:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.292697:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27254784, 27258880) 0 0 00000080:00200000:0.0:1789760784.292701:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.292714:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6654, 6654] 00020000:00200000:0.0:1789760784.292721:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27254784 chunk: [27254784, 27258880) 27258880, 4096 00020000:00200000:0.0:1789760784.292725:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.292742:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6654, 6654] 00000008:00010000:0.0:1789760784.292755:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.292760:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.292769:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27254784, 27258880) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.292779:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27254784=0x19fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.292823:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.292826:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27258880 00000020:00010000:0.0:1789760784.292833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.292836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.292839:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.292843:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.292846:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.292864:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.292871:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.292885:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760784.292976:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 27254784<>27258880 00000100:00100000:1.0:1789760784.293933:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495935488:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760784.377920:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760784.378116:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.378126:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.378144:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760784.378199:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495935488:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.378435:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6654, 6654] 00000080:00200000:0.0:1789760784.378448:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.378453:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.378464:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27258880 i_blocks 53257 00000020:00200000:0.0:1789760784.378468:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.378476:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.378500:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27258880, count: 4096 00000080:00200000:0.0:1789760784.378506:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27258880, bytes: 4096 00000080:00200000:0.0:1789760784.378514:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.378522:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.378524:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27258880, 27262976) 0 0 00000080:00200000:0.0:1789760784.378527:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.378540:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6655, 6655] 00020000:00200000:0.0:1789760784.378547:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27258880 chunk: [27258880, 27262976) 27262976, 4096 00020000:00200000:0.0:1789760784.378551:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.378565:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6655, 6655] 00000008:00010000:0.0:1789760784.378577:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.378582:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.378590:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27258880, 27262976) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.378599:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27258880=0x19ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.378713:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.378716:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27262976 00000020:00010000:0.0:1789760784.378722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.378726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.378728:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.378732:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.378735:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.378741:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.378747:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.378759:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760784.378930:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 27258880<>27262976 00000100:00100000:1.0:1789760784.379019:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760784.379035:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495935616:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.468557:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973972 gen 2 00000008:00010000:1.0:1789760784.468605:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760784.468712:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.468722:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.468779:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760784.468806:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495935616:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.468940:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6655, 6655] 00000080:00200000:0.0:1789760784.468953:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.468958:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.468970:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27262976 i_blocks 53257 00000020:00200000:0.0:1789760784.468973:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.468982:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.469008:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27262976, count: 4096 00000080:00200000:0.0:1789760784.469014:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27262976, bytes: 4096 00000080:00200000:0.0:1789760784.469021:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.469028:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.469031:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27262976, 27267072) 0 0 00000080:00200000:0.0:1789760784.469034:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.469048:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6656, 6656] 00020000:00200000:0.0:1789760784.469055:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27262976 chunk: [27262976, 27267072) 27267072, 4096 00020000:00200000:0.0:1789760784.469059:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.469073:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6656, 6656] 00000008:00010000:0.0:1789760784.469087:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.469092:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.469101:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27262976, 27267072) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.469111:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27262976=0x1a00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.469154:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.469157:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27267072 00000020:00010000:0.0:1789760784.469164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.469244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.469249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.469253:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.469255:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.469263:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.469412:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.469431:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760784.469528:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 27262976<>27267072 00000100:00100000:1.0:1789760784.469585:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495935744:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.516550:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973973 gen 2 00000008:00010000:1.0:1789760784.516583:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760784.516590:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.516598:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.516621:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760784.516643:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495935744:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.516782:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6656, 6656] 00000080:00200000:0.0:1789760784.516794:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.516799:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.516810:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27267072 i_blocks 53257 00000020:00200000:0.0:1789760784.516813:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.516820:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.516845:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27267072, count: 4096 00000080:00200000:0.0:1789760784.516859:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27267072, bytes: 4096 00000080:00200000:0.0:1789760784.516869:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.516876:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.516878:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27267072, 27271168) 0 0 00000080:00200000:0.0:1789760784.516881:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.516894:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6657, 6657] 00020000:00200000:0.0:1789760784.516901:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27267072 chunk: [27267072, 27271168) 27271168, 4096 00020000:00200000:0.0:1789760784.516905:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.516919:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6657, 6657] 00000008:00010000:0.0:1789760784.516934:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.516940:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.516949:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27267072, 27271168) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.516959:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27267072=0x1a01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.517001:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.517004:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27271168 00000020:00010000:0.0:1789760784.517009:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.517012:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.517015:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.517019:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.517022:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.517027:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.517033:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.517045:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760784.517133:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 27267072<>27271168 00000100:00100000:1.0:1789760784.517359:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495935872:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.577240:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973974 gen 2 00000008:00010000:1.0:1789760784.577275:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760784.577282:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.577302:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.577323:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760784.577349:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495935872:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.577440:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6657, 6657] 00000080:00200000:0.0:1789760784.577452:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.577457:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.577468:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27271168 i_blocks 55305 00000020:00200000:0.0:1789760784.577472:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.577480:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.577504:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27271168, count: 4096 00000080:00200000:0.0:1789760784.577509:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27271168, bytes: 4096 00000080:00200000:0.0:1789760784.577517:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.577524:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.577527:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27271168, 27275264) 0 0 00000080:00200000:0.0:1789760784.577530:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.577542:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6658, 6658] 00020000:00200000:0.0:1789760784.577550:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27271168 chunk: [27271168, 27275264) 27275264, 4096 00020000:00200000:0.0:1789760784.577553:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.577567:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6658, 6658] 00000008:00010000:0.0:1789760784.577582:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.577587:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.577596:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27271168, 27275264) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.577606:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27271168=0x1a02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.577649:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.577652:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27275264 00000020:00010000:0.0:1789760784.577659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.577662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.577664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.577668:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.577671:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.577677:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.577683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.577696:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760784.577789:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 27271168<>27275264 00000100:00100000:1.0:1789760784.577879:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760784.577891:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495936000:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760784.642213:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760784.642224:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.642233:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.642250:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760784.642274:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495936000:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.642365:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6658, 6658] 00000080:00200000:0.0:1789760784.642377:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.642382:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.642393:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27275264 i_blocks 55305 00000020:00200000:0.0:1789760784.642396:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.642404:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.642427:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27275264, count: 4096 00000080:00200000:0.0:1789760784.642433:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27275264, bytes: 4096 00000080:00200000:0.0:1789760784.642440:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.642447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.642449:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27275264, 27279360) 0 0 00000080:00200000:0.0:1789760784.642452:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.642465:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6659, 6659] 00020000:00200000:0.0:1789760784.642473:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27275264 chunk: [27275264, 27279360) 27279360, 4096 00020000:00200000:0.0:1789760784.642476:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.642490:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6659, 6659] 00000008:00010000:0.0:1789760784.642503:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.642508:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.642516:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27275264, 27279360) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.642526:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27275264=0x1a03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.642568:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.642571:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27279360 00000020:00010000:0.0:1789760784.642577:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.642581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.642583:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.642587:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.642589:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.642596:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.642601:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.642684:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760784.642783:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27275264<>27279360 00000100:00100000:1.0:1789760784.642904:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495936128:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.720926:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973976 gen 2 00000008:00010000:1.0:1789760784.720971:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760784.720977:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.720985:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.721008:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760784.721041:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495936128:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.721220:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6659, 6659] 00000080:00200000:0.0:1789760784.721233:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.721238:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.721250:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27279360 i_blocks 55305 00000020:00200000:0.0:1789760784.721253:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.721262:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.721302:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27279360, count: 4096 00000080:00200000:0.0:1789760784.721309:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27279360, bytes: 4096 00000080:00200000:0.0:1789760784.721320:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.721326:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.721328:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27279360, 27283456) 0 0 00000080:00200000:0.0:1789760784.721332:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.721345:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6660, 6660] 00020000:00200000:0.0:1789760784.721352:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27279360 chunk: [27279360, 27283456) 27283456, 4096 00020000:00200000:0.0:1789760784.721356:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.721370:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6660, 6660] 00000008:00010000:0.0:1789760784.721383:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.721388:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.721397:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27279360, 27283456) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.721408:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27279360=0x1a04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.721453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.721456:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27283456 00000020:00010000:0.0:1789760784.721462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.721465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.721468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.721472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.721474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.721480:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.721486:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.721500:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760784.721599:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 27279360<>27283456 00000100:00100000:1.0:1789760784.721709:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495936256:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760784.770753:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760784.770764:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.770774:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.770791:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760784.770815:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495936256:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.771110:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6660, 6660] 00000080:00200000:0.0:1789760784.771123:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.771129:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.771140:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27283456 i_blocks 55305 00000020:00200000:0.0:1789760784.771143:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.771150:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.771199:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27283456, count: 4096 00000080:00200000:0.0:1789760784.771206:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27283456, bytes: 4096 00000080:00200000:0.0:1789760784.771215:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.771222:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.771224:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27283456, 27287552) 0 0 00000080:00200000:0.0:1789760784.771228:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.771240:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6661, 6661] 00020000:00200000:0.0:1789760784.771247:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27283456 chunk: [27283456, 27287552) 27287552, 4096 00020000:00200000:0.0:1789760784.771250:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.771264:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6661, 6661] 00000008:00010000:0.0:1789760784.771277:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.771282:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.771302:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27283456, 27287552) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.771314:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27283456=0x1a05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.771357:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.771360:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27287552 00000020:00010000:0.0:1789760784.771367:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.771370:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.771373:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.771377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.771380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.771386:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.771391:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.771405:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760784.771503:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 27283456<>27287552 00000100:00100000:1.0:1789760784.772771:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760784.772796:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495936384:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.814241:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973978 gen 2 00000008:00010000:1.0:1789760784.814298:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760784.814306:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.814314:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.814336:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760784.814360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495936384:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.814459:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6661, 6661] 00000080:00200000:0.0:1789760784.814472:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.814477:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.814488:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27287552 i_blocks 55305 00000020:00200000:0.0:1789760784.814492:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.814500:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.814525:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27287552, count: 4096 00000080:00200000:0.0:1789760784.814531:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27287552, bytes: 4096 00000080:00200000:0.0:1789760784.814550:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.814557:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.814560:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27287552, 27291648) 0 0 00000080:00200000:0.0:1789760784.814563:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.814574:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6662, 6662] 00020000:00200000:0.0:1789760784.814581:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27287552 chunk: [27287552, 27291648) 27291648, 4096 00020000:00200000:0.0:1789760784.814584:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.814599:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6662, 6662] 00000008:00010000:0.0:1789760784.814614:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.814619:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.814627:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27287552, 27291648) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.814636:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27287552=0x1a06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.814680:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.814683:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27291648 00000020:00010000:0.0:1789760784.814691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.814694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.814697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.814701:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.814703:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.814710:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.814715:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.814728:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760784.814833:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 27287552<>27291648 00000100:00100000:1.0:1789760784.815161:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495936512:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760784.844411:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760784.871614:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973979 gen 2 00000008:00010000:1.0:1789760784.871650:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760784.871657:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.871667:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.871734:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760784.871757:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495936512:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760784.877728:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6662, 6662] 00000080:00200000:0.0:1789760784.877749:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.877755:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.877770:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27291648 i_blocks 55305 00000020:00200000:0.0:1789760784.877774:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.877786:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.877812:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27291648, count: 4096 00000080:00200000:0.0:1789760784.877819:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27291648, bytes: 4096 00000080:00200000:0.0:1789760784.877830:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.877837:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.877839:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27291648, 27295744) 0 0 00000080:00200000:0.0:1789760784.877843:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.877856:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6663, 6663] 00020000:00200000:0.0:1789760784.877862:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27291648 chunk: [27291648, 27295744) 27295744, 4096 00020000:00200000:0.0:1789760784.877866:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.877882:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6663, 6663] 00000008:00010000:0.0:1789760784.877895:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.877899:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760784.877908:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27291648, 27295744) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.877917:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27291648=0x1a07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.877968:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.877971:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27295744 00000020:00010000:0.0:1789760784.877977:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.877980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.877983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.877986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.877989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.877996:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.878001:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.878128:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760784.880890:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 27291648<>27295744 00000100:00100000:1.0:1789760784.883337:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760784.883367:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495936640:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760784.959643:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973980 gen 2 00000008:00010000:1.0:1789760784.959681:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760784.959687:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.959696:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.959716:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760784.959739:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495936640:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760784.959890:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6663, 6663] 00000080:00200000:2.0:1789760784.959902:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.959907:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.959918:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27295744 i_blocks 55305 00000020:00200000:2.0:1789760784.959921:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.959929:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.959954:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27295744, count: 4096 00000080:00200000:2.0:1789760784.959960:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27295744, bytes: 4096 00000080:00200000:2.0:1789760784.959968:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.959974:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.959977:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27295744, 27299840) 0 0 00000080:00200000:2.0:1789760784.959980:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.959993:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6664, 6664] 00020000:00200000:2.0:1789760784.960000:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27295744 chunk: [27295744, 27299840) 27299840, 4096 00020000:00200000:2.0:1789760784.960004:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.960031:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6664, 6664] 00000008:00010000:2.0:1789760784.960044:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.960049:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760784.960058:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27295744, 27299840) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.960069:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27295744=0x1a08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.960112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.960115:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27299840 00000020:00010000:2.0:1789760784.960121:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.960125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.960127:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.960131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088148a8 (ffff8dd205fe0d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.960133:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.960140:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.960145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.960158:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760784.960300:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 27295744<>27299840 00000100:00100000:2.0:1789760784.960338:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760784.960354:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495936768:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760785.017231:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973981 gen 2 00000008:00010000:2.0:1789760785.017267:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760785.017274:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.017296:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760785.017317:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760785.017341:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495936768:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.017449:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6664, 6664] 00000080:00200000:3.0:1789760785.017461:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.017468:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.017481:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27299840 i_blocks 55305 00000020:00200000:3.0:1789760785.017485:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.017493:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.017542:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27299840, count: 4096 00000080:00200000:3.0:1789760785.017549:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27299840, bytes: 4096 00000080:00200000:3.0:1789760785.017558:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.017564:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.017566:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27299840, 27303936) 0 0 00000080:00200000:3.0:1789760785.017570:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.017583:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6665, 6665] 00020000:00200000:3.0:1789760785.017607:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27299840 chunk: [27299840, 27303936) 27303936, 4096 00020000:00200000:3.0:1789760785.017612:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.017627:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6665, 6665] 00000008:00010000:3.0:1789760785.017639:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.017644:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.017653:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27299840, 27303936) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.017887:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27299840=0x1a09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.017937:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.017940:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27303936 00000020:00010000:3.0:1789760785.017947:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.017951:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.017954:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.017957:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.017959:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.017966:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.017972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.017988:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760785.018082:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 27299840<>27303936 00000100:00100000:3.0:1789760785.018121:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760785.018136:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495936896:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760785.081880:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973982 gen 2 00000008:00010000:3.0:1789760785.081919:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:3.0:1789760785.081925:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.081934:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.081957:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:3.0:1789760785.081982:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495936896:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760785.082082:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6665, 6665] 00000080:00200000:2.0:1789760785.082095:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.082101:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.082113:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27303936 i_blocks 55305 00000020:00200000:2.0:1789760785.082117:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.082125:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.082149:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27303936, count: 4096 00000080:00200000:2.0:1789760785.082156:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27303936, bytes: 4096 00000080:00200000:2.0:1789760785.082164:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.082207:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.082210:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27303936, 27308032) 0 0 00000080:00200000:2.0:1789760785.082213:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.082239:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6666, 6666] 00020000:00200000:2.0:1789760785.082246:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27303936 chunk: [27303936, 27308032) 27308032, 4096 00020000:00200000:2.0:1789760785.082250:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.082266:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6666, 6666] 00000008:00010000:2.0:1789760785.082292:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.082297:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760785.082306:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27303936, 27308032) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.082317:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27303936=0x1a0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.082360:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.082363:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27308032 00000020:00010000:2.0:1789760785.082370:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.082373:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.082376:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.082380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.082382:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.082387:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.082393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.082404:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760785.082498:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 27303936<>27308032 00000100:00100000:2.0:1789760785.082538:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760785.082552:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495937024:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760785.145818:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973983 gen 2 00000008:00010000:2.0:1789760785.145868:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760785.145875:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.145884:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760785.145906:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760785.145930:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495937024:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.146416:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6666, 6666] 00000080:00200000:3.0:1789760785.146430:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.146436:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.146449:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27308032 i_blocks 55305 00000020:00200000:3.0:1789760785.146453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.146462:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.146488:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27308032, count: 4096 00000080:00200000:3.0:1789760785.146495:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27308032, bytes: 4096 00000080:00200000:3.0:1789760785.146504:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.146511:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.146514:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27308032, 27312128) 0 0 00000080:00200000:3.0:1789760785.146517:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.146531:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6667, 6667] 00020000:00200000:3.0:1789760785.146538:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27308032 chunk: [27308032, 27312128) 27312128, 4096 00020000:00200000:3.0:1789760785.146542:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.146560:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6667, 6667] 00000008:00010000:3.0:1789760785.146574:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.146580:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.146666:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27308032, 27312128) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.146679:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27308032=0x1a0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.146728:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.146731:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27312128 00000020:00010000:3.0:1789760785.146739:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.146742:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.146746:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.146749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.146752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.146758:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.146764:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.146779:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760785.146947:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 27308032<>27312128 00000100:00100000:2.0:1789760785.147009:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495937152:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760785.184801:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973984 gen 2 00000008:00010000:2.0:1789760785.184893:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:2.0:1789760785.184900:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.184910:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760785.184924:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:2.0:1789760785.184949:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495937152:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.185542:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6667, 6667] 00000080:00200000:3.0:1789760785.185590:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.185596:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.185617:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27312128 i_blocks 55305 00000020:00200000:3.0:1789760785.185621:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.185630:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.185656:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27312128, count: 4096 00000080:00200000:3.0:1789760785.185662:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27312128, bytes: 4096 00000080:00200000:3.0:1789760785.185672:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.185679:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.185681:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27312128, 27316224) 0 0 00000080:00200000:3.0:1789760785.185684:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.185698:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6668, 6668] 00020000:00200000:3.0:1789760785.185705:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27312128 chunk: [27312128, 27316224) 27316224, 4096 00020000:00200000:3.0:1789760785.185708:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.185723:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6668, 6668] 00000008:00010000:3.0:1789760785.185736:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.185743:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.185752:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27312128, 27316224) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.185763:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27312128=0x1a0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.185812:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.185814:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27316224 00000020:00010000:3.0:1789760785.185821:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.185824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.185827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.185830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.185833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.185839:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.185844:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.185869:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082506c0 00000008:00100000:3.0:1789760785.185966:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 27312128<>27316224 00000100:00100000:2.0:1789760785.186138:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495937280:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760785.237656:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082506c0 00000020:00200000:2.0:1789760785.237667:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.237677:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760785.237696:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082506c0 00000100:00100000:2.0:1789760785.237721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495937280:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.237911:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6668, 6668] 00000080:00200000:3.0:1789760785.237924:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.237929:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.237941:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27316224 i_blocks 55305 00000020:00200000:3.0:1789760785.237945:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.237953:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.237978:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27316224, count: 4096 00000080:00200000:3.0:1789760785.237984:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27316224, bytes: 4096 00000080:00200000:3.0:1789760785.237991:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.237998:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.238000:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27316224, 27320320) 0 0 00000080:00200000:3.0:1789760785.238004:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.238016:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6669, 6669] 00020000:00200000:3.0:1789760785.238023:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27316224 chunk: [27316224, 27320320) 27320320, 4096 00020000:00200000:3.0:1789760785.238028:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.238044:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6669, 6669] 00000008:00010000:3.0:1789760785.238058:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.238064:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.238072:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27316224, 27320320) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.238082:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27316224=0x1a0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.238127:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.238129:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27320320 00000020:00010000:3.0:1789760785.238136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.238139:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.238142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.238145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.238147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.238154:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.238159:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.238211:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082509c0 00000008:00100000:3.0:1789760785.238320:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 27316224<>27320320 00000100:00100000:2.0:1789760785.239490:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760785.239514:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495937408:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760785.284244:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973985 gen 2 00000100:00100000:2.0:1789760785.284277:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203146300 x1876699495937408/t4294973986(4294973986) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760801 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760785.284328:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082509c0 00000020:00200000:2.0:1789760785.284344:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.284352:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760785.284373:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082509c0 00000100:00100000:2.0:1789760785.284394:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495937408:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.284546:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6669, 6669] 00000080:00200000:3.0:1789760785.284559:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.284564:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.284577:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27320320 i_blocks 55305 00000020:00200000:3.0:1789760785.284580:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.284587:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.284624:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27320320, count: 4096 00000080:00200000:3.0:1789760785.284631:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27320320, bytes: 4096 00000080:00200000:3.0:1789760785.284638:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.284646:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.284648:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27320320, 27324416) 0 0 00000080:00200000:3.0:1789760785.284652:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.284666:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6670, 6670] 00020000:00200000:3.0:1789760785.284674:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27320320 chunk: [27320320, 27324416) 27324416, 4096 00020000:00200000:3.0:1789760785.284678:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.284694:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6670, 6670] 00000008:00010000:3.0:1789760785.284708:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.284713:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.284722:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27320320, 27324416) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.284734:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27320320=0x1a0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.284776:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.284779:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27324416 00000020:00010000:3.0:1789760785.284786:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.284789:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.284793:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.284797:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.284799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.284806:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.284812:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.284827:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250540 00000008:00100000:3.0:1789760785.284925:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 27320320<>27324416 00000100:00100000:2.0:1789760785.285024:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760785.285040:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495937536:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760785.337493:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973986 gen 2 00000100:00100000:2.0:1789760785.337529:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203144a80 x1876699495937536/t4294973987(4294973987) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760801 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760785.337567:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250540 00000020:00200000:2.0:1789760785.337573:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.337582:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.337693:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6670, 6670] 00000080:00200000:3.0:1789760785.337708:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.337713:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.337728:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27324416 i_blocks 55305 00000020:00200000:3.0:1789760785.337732:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.337740:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.337766:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27324416, count: 4096 00000080:00200000:3.0:1789760785.337773:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27324416, bytes: 4096 00000080:00200000:3.0:1789760785.337781:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.337788:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.337790:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27324416, 27328512) 0 0 00000080:00200000:3.0:1789760785.337795:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.337807:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6671, 6671] 00020000:00200000:3.0:1789760785.337816:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27324416 chunk: [27324416, 27328512) 27328512, 4096 00020000:00200000:3.0:1789760785.337819:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.337834:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6671, 6671] 00000008:00010000:3.0:1789760785.337861:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.337867:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.337878:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27324416, 27328512) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.337889:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27324416=0x1a0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.337939:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.337941:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27328512 00000020:00010000:3.0:1789760785.337949:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.337952:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.337955:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.337958:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.337960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.337968:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.337973:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.337990:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760785.338081:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 27324416<>27328512 00000100:00100000:3.0:1789760785.338126:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760785.338142:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495937664:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760785.338832:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250540 00000100:00100000:2.0:1789760785.338857:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495937536:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760785.377739:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973988 gen 2 00000008:00010000:3.0:1789760785.377785:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:3.0:1789760785.377791:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.377800:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.377820:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:3.0:1789760785.377845:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495937664:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760785.377975:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6671, 6671] 00000080:00200000:2.0:1789760785.377987:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.377992:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.378004:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27328512 i_blocks 55305 00000020:00200000:2.0:1789760785.378008:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.378017:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.378046:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27328512, count: 4096 00000080:00200000:2.0:1789760785.378051:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27328512, bytes: 4096 00000080:00200000:2.0:1789760785.378060:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.378067:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.378069:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27328512, 27332608) 0 0 00000080:00200000:2.0:1789760785.378072:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.378085:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6672, 6672] 00020000:00200000:2.0:1789760785.378092:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27328512 chunk: [27328512, 27332608) 27332608, 4096 00020000:00200000:2.0:1789760785.378096:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.378112:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6672, 6672] 00000008:00010000:2.0:1789760785.378127:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.378133:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760785.378142:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27328512, 27332608) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.378153:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27328512=0x1a10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.378227:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.378230:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27332608 00000020:00010000:2.0:1789760785.378237:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.378241:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.378244:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.378247:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.378249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.378395:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.378402:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.378417:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760785.378521:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 27328512<>27332608 00000100:00100000:3.0:1789760785.378584:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495937792:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760785.456073:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973989 gen 2 00000008:00010000:3.0:1789760785.456113:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760785.456120:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.456129:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.456154:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760785.456211:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495937792:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760785.456402:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6672, 6672] 00000080:00200000:2.0:1789760785.456413:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.456419:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.456431:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27332608 i_blocks 55305 00000020:00200000:2.0:1789760785.456434:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.456442:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.456468:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27332608, count: 4096 00000080:00200000:2.0:1789760785.456474:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27332608, bytes: 4096 00000080:00200000:2.0:1789760785.456481:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.456488:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.456491:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27332608, 27336704) 0 0 00000080:00200000:2.0:1789760785.456494:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.456506:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6673, 6673] 00020000:00200000:2.0:1789760785.456513:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27332608 chunk: [27332608, 27336704) 27336704, 4096 00020000:00200000:2.0:1789760785.456517:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.456530:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6673, 6673] 00000008:00010000:2.0:1789760785.456543:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.456548:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760785.456556:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27332608, 27336704) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.456567:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27332608=0x1a11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.456672:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.456675:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27336704 00000020:00010000:2.0:1789760785.456683:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.456686:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.456689:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.456692:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.456695:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.456701:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.456706:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.456719:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760785.458504:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 27332608<>27336704 00000100:00100000:3.0:1789760785.458597:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495937920:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760785.526323:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973990 gen 2 00000008:00010000:3.0:1789760785.526429:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760785.526436:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.526446:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.526466:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760785.526492:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495937920:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760785.526587:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6673, 6673] 00000080:00200000:0.0:1789760785.526607:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.526613:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.526626:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27336704 i_blocks 55305 00000020:00200000:0.0:1789760785.526629:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.526637:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.526662:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27336704, count: 4096 00000080:00200000:0.0:1789760785.526668:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27336704, bytes: 4096 00000080:00200000:0.0:1789760785.526676:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.526683:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.526685:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27336704, 27340800) 0 0 00000080:00200000:0.0:1789760785.526689:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.526702:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6674, 6674] 00020000:00200000:0.0:1789760785.526709:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27336704 chunk: [27336704, 27340800) 27340800, 4096 00020000:00200000:0.0:1789760785.526713:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.526729:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6674, 6674] 00000008:00010000:0.0:1789760785.526743:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.526748:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760785.526757:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27336704, 27340800) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.526767:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27336704=0x1a12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.526813:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.526816:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27340800 00000020:00010000:0.0:1789760785.526824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.526827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.526830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.526833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.526836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.526842:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.526864:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.526879:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760785.527643:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 27336704<>27340800 00000100:00100000:1.0:1789760785.527704:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495938048:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760785.582914:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973991 gen 2 00000008:00010000:1.0:1789760785.582944:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760785.582949:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.582956:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.582972:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760785.582991:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495938048:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.583081:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6674, 6674] 00000080:00200000:3.0:1789760785.583092:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.583096:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.583106:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27340800 i_blocks 55305 00000020:00200000:3.0:1789760785.583108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.583114:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.583136:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27340800, count: 4096 00000080:00200000:3.0:1789760785.583141:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27340800, bytes: 4096 00000080:00200000:3.0:1789760785.583147:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.583152:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.583153:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27340800, 27344896) 0 0 00000080:00200000:3.0:1789760785.583156:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.583166:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6675, 6675] 00020000:00200000:3.0:1789760785.583194:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27340800 chunk: [27340800, 27344896) 27344896, 4096 00020000:00200000:3.0:1789760785.583197:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.583210:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6675, 6675] 00000008:00010000:3.0:1789760785.583223:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.583228:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.583236:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27340800, 27344896) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.583246:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27340800=0x1a13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.583305:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.583307:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27344896 00000020:00010000:3.0:1789760785.583313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.583317:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.583319:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.583322:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.583324:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.583407:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.583413:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.583427:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760785.583521:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 27340800<>27344896 00000100:00100000:3.0:1789760785.583562:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760785.583576:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495938176:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760785.632664:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:3.0:1789760785.632675:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.632685:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.632705:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:3.0:1789760785.632727:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495938176:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760785.632857:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6675, 6675] 00000080:00200000:1.0:1789760785.632870:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.632875:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.632887:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27344896 i_blocks 55305 00000020:00200000:1.0:1789760785.632891:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.632898:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.632923:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27344896, count: 4096 00000080:00200000:1.0:1789760785.632929:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27344896, bytes: 4096 00000080:00200000:1.0:1789760785.632937:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.632943:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.632945:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27344896, 27348992) 0 0 00000080:00200000:1.0:1789760785.632949:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.632962:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6676, 6676] 00020000:00200000:1.0:1789760785.632969:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27344896 chunk: [27344896, 27348992) 27348992, 4096 00020000:00200000:1.0:1789760785.632973:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.632989:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6676, 6676] 00000008:00010000:1.0:1789760785.633004:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.633009:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760785.633018:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27344896, 27348992) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.633027:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27344896=0x1a14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.633069:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.633072:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27348992 00000020:00010000:1.0:1789760785.633079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.633082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.633085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.633088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.633091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.633097:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.633103:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.633117:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760785.633244:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27344896<>27348992 00000100:00100000:1.0:1789760785.633289:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760785.633302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495938304:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760785.677578:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973993 gen 2 00000008:00010000:1.0:1789760785.677736:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760785.677743:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.677751:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.677773:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760785.677797:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495938304:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.677887:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6676, 6676] 00000080:00200000:3.0:1789760785.677900:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.677904:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.677916:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27348992 i_blocks 55305 00000020:00200000:3.0:1789760785.677920:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.677927:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.677952:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27348992, count: 4096 00000080:00200000:3.0:1789760785.677958:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27348992, bytes: 4096 00000080:00200000:3.0:1789760785.677966:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.677972:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.677974:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27348992, 27353088) 0 0 00000080:00200000:3.0:1789760785.677978:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.677990:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6677, 6677] 00020000:00200000:3.0:1789760785.677998:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27348992 chunk: [27348992, 27353088) 27353088, 4096 00020000:00200000:3.0:1789760785.678001:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.678051:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6677, 6677] 00000008:00010000:3.0:1789760785.678065:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.678070:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.678078:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27348992, 27353088) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.678089:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27348992=0x1a15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.678134:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.678137:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27353088 00000020:00010000:3.0:1789760785.678143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.678146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.678149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.678153:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.678155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.678161:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.678617:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.678740:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250240 00000008:00100000:3.0:1789760785.678838:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 27348992<>27353088 00000100:00100000:2.0:1789760785.678952:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495938432:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760785.720982:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250240 00000020:00200000:2.0:1789760785.720994:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.721003:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760785.721025:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250240 00000100:00100000:2.0:1789760785.721050:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495938432:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.721138:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6677, 6677] 00000080:00200000:3.0:1789760785.721152:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.721158:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.721197:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27353088 i_blocks 55305 00000020:00200000:3.0:1789760785.721201:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.721209:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.721234:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27353088, count: 4096 00000080:00200000:3.0:1789760785.721240:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27353088, bytes: 4096 00000080:00200000:3.0:1789760785.721248:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.721266:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.721268:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27353088, 27357184) 0 0 00000080:00200000:3.0:1789760785.721273:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.721294:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6678, 6678] 00020000:00200000:3.0:1789760785.721300:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27353088 chunk: [27353088, 27357184) 27357184, 4096 00020000:00200000:3.0:1789760785.721304:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.721321:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6678, 6678] 00000008:00010000:3.0:1789760785.721333:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.721338:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.721347:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27353088, 27357184) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.721357:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27353088=0x1a16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.721400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.721403:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27357184 00000020:00010000:3.0:1789760785.721410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.721413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.721416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.721419:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.721421:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.721427:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.721433:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.721446:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760785.721545:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 27353088<>27357184 00000100:00100000:2.0:1789760785.722372:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760785.722392:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495938560:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760785.755466:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973994 gen 2 00000100:00100000:3.0:1789760785.755501:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203145180 x1876699495938560/t4294973995(4294973995) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760801 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760785.755541:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:3.0:1789760785.755547:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.755555:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.755564:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:3.0:1789760785.755582:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495938560:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760785.755832:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6678, 6678] 00000080:00200000:1.0:1789760785.755846:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.755863:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.755876:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27357184 i_blocks 55305 00000020:00200000:1.0:1789760785.755880:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.755888:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.755916:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27357184, count: 4096 00000080:00200000:1.0:1789760785.755922:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27357184, bytes: 4096 00000080:00200000:1.0:1789760785.755932:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.755939:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.755941:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27357184, 27361280) 0 0 00000080:00200000:1.0:1789760785.755945:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.755959:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6679, 6679] 00020000:00200000:1.0:1789760785.755967:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27357184 chunk: [27357184, 27361280) 27361280, 4096 00020000:00200000:1.0:1789760785.755971:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.755986:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6679, 6679] 00000008:00010000:1.0:1789760785.755999:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.756004:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760785.756013:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27357184, 27361280) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.756024:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27357184=0x1a17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.756071:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.756075:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27361280 00000020:00010000:1.0:1789760785.756082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.756085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.756089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.756092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.756095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.756101:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.756106:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.756119:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760785.756412:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27357184<>27361280 00000100:00100000:0.0:1789760785.756503:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495938688:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760785.793209:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973996 gen 2 00000008:00010000:1.0:1789760785.793267:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760785.793274:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.793291:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.793310:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760785.793340:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495938688:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.793537:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6679, 6679] 00000080:00200000:3.0:1789760785.793550:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.793556:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.793568:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27361280 i_blocks 55305 00000020:00200000:3.0:1789760785.793572:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.793580:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.793674:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27361280, count: 4096 00000080:00200000:3.0:1789760785.793682:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27361280, bytes: 4096 00000080:00200000:3.0:1789760785.793692:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.793701:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.793703:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27361280, 27365376) 0 0 00000080:00200000:3.0:1789760785.793707:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.793720:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6680, 6680] 00020000:00200000:3.0:1789760785.793729:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27361280 chunk: [27361280, 27365376) 27365376, 4096 00020000:00200000:3.0:1789760785.793733:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.793748:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6680, 6680] 00000008:00010000:3.0:1789760785.793760:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.793765:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.793773:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27361280, 27365376) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.793785:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27361280=0x1a18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.793828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.793830:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27365376 00000020:00010000:3.0:1789760785.793837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.793840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.793842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.793845:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.793848:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.793853:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.793859:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.793873:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760785.793969:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 27361280<>27365376 00000100:00100000:2.0:1789760785.794074:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495938816:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760785.840510:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:2.0:1789760785.840521:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.840530:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760785.840547:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:2.0:1789760785.840570:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495938816:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760785.840660:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6680, 6680] 00000080:00200000:3.0:1789760785.840672:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.840677:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.840688:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27365376 i_blocks 55305 00000020:00200000:3.0:1789760785.840692:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.840699:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.840722:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27365376, count: 4096 00000080:00200000:3.0:1789760785.840728:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27365376, bytes: 4096 00000080:00200000:3.0:1789760785.840736:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.840743:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.840745:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27365376, 27369472) 0 0 00000080:00200000:3.0:1789760785.840749:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.840761:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6681, 6681] 00020000:00200000:3.0:1789760785.840768:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27365376 chunk: [27365376, 27369472) 27369472, 4096 00020000:00200000:3.0:1789760785.840772:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.840786:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6681, 6681] 00000008:00010000:3.0:1789760785.840800:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.840805:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.840814:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27365376, 27369472) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.840825:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27365376=0x1a19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.840903:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.840906:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27369472 00000020:00010000:3.0:1789760785.840912:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.840915:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.840918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.840922:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.840924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.840931:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.840935:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.840949:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760785.841039:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 27365376<>27369472 00000100:00100000:2.0:1789760785.841456:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495938944:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760785.867922:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760785.891226:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973998 gen 2 00000008:00010000:2.0:1789760785.891273:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760785.891293:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760785.891311:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.891621:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6681, 6681] 00000080:00200000:3.0:1789760785.891636:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.891641:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.891652:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27369472 i_blocks 55305 00000020:00200000:3.0:1789760785.891656:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.891664:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.891687:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27369472, count: 4096 00000080:00200000:3.0:1789760785.891694:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27369472, bytes: 4096 00000080:00200000:3.0:1789760785.891703:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.891712:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.891714:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27369472, 27373568) 0 0 00000080:00200000:3.0:1789760785.891718:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.891732:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6682, 6682] 00020000:00200000:3.0:1789760785.891740:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27369472 chunk: [27369472, 27373568) 27373568, 4096 00020000:00200000:3.0:1789760785.891744:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.891760:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6682, 6682] 00000008:00010000:3.0:1789760785.891775:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.891781:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760785.891791:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27369472, 27373568) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.891806:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27369472=0x1a1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.891867:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.891869:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27373568 00000020:00010000:3.0:1789760785.891875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.891878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.891881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.891884:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.891887:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.891893:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.891898:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.891912:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760785.892000:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 27369472<>27373568 00000100:00100000:3.0:1789760785.892056:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495939072:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760785.892796:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760785.892818:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495938944:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760785.950821:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294973999 gen 2 00000008:00010000:3.0:1789760785.950874:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:3.0:1789760785.950882:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.950891:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.950914:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:3.0:1789760785.950940:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495939072:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760785.951152:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6682, 6682] 00000080:00200000:1.0:1789760785.951166:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.951201:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.951214:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27373568 i_blocks 55305 00000020:00200000:1.0:1789760785.951218:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.951227:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.951252:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27373568, count: 4096 00000080:00200000:1.0:1789760785.951257:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27373568, bytes: 4096 00000080:00200000:1.0:1789760785.951265:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.951273:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.951284:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27373568, 27377664) 0 0 00000080:00200000:1.0:1789760785.951288:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.951300:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6683, 6683] 00020000:00200000:1.0:1789760785.951307:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27373568 chunk: [27373568, 27377664) 27377664, 4096 00020000:00200000:1.0:1789760785.951310:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.951395:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6683, 6683] 00000008:00010000:1.0:1789760785.951410:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.951415:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760785.951425:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27373568, 27377664) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.951436:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27373568=0x1a1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.951480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.951482:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27377664 00000020:00010000:1.0:1789760785.951489:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.951492:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.951495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.951498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.951501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.951507:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.951512:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.951528:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760785.951630:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27373568<>27377664 00000100:00100000:1.0:1789760785.951670:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760785.951686:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495939200:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.015614:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974000 gen 2 00000008:00010000:1.0:1789760786.015652:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760786.015660:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.015669:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.015687:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760786.015710:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495939200:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.015870:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6683, 6683] 00000080:00200000:0.0:1789760786.015884:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.015889:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.015901:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27377664 i_blocks 55305 00000020:00200000:0.0:1789760786.015906:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.015913:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.015940:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27377664, count: 4096 00000080:00200000:0.0:1789760786.015946:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27377664, bytes: 4096 00000080:00200000:0.0:1789760786.015955:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.015963:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.015965:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27377664, 27381760) 0 0 00000080:00200000:0.0:1789760786.015969:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.015982:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6684, 6684] 00020000:00200000:0.0:1789760786.015989:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27377664 chunk: [27377664, 27381760) 27381760, 4096 00020000:00200000:0.0:1789760786.015993:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.016009:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6684, 6684] 00000008:00010000:0.0:1789760786.016024:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.016029:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.016037:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27377664, 27381760) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.016050:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27377664=0x1a1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.016098:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.016101:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27381760 00000020:00010000:0.0:1789760786.016108:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.016112:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.016114:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.016117:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.016120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.016125:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.016132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.016160:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760786.016315:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 27377664<>27381760 00000100:00100000:0.0:1789760786.016688:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760786.016705:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495939328:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.055022:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974001 gen 2 00000008:00010000:1.0:1789760786.055061:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760786.055067:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.055075:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.055097:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760786.055122:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495939328:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.055309:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6684, 6684] 00000080:00200000:0.0:1789760786.055321:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.055335:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.055347:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27381760 i_blocks 55305 00000020:00200000:0.0:1789760786.055350:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.055358:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.055382:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27381760, count: 4096 00000080:00200000:0.0:1789760786.055387:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27381760, bytes: 4096 00000080:00200000:0.0:1789760786.055394:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.055401:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.055403:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27381760, 27385856) 0 0 00000080:00200000:0.0:1789760786.055407:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.055419:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6685, 6685] 00020000:00200000:0.0:1789760786.055426:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27381760 chunk: [27381760, 27385856) 27385856, 4096 00020000:00200000:0.0:1789760786.055429:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.055445:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6685, 6685] 00000008:00010000:0.0:1789760786.055459:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.055464:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.055473:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27381760, 27385856) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.055483:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27381760=0x1a1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.055528:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.055531:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27385856 00000020:00010000:0.0:1789760786.055538:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.055541:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.055544:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.055547:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.055550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.055555:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.055561:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.055574:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760786.055679:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27381760<>27385856 00000100:00100000:1.0:1789760786.055865:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760786.055882:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495939456:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760786.108355:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760786.108365:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.108374:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.108393:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760786.108416:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495939456:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.108558:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6685, 6685] 00000080:00200000:0.0:1789760786.108572:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.108578:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.108590:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27385856 i_blocks 55305 00000020:00200000:0.0:1789760786.108604:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.108613:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.108641:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27385856, count: 4096 00000080:00200000:0.0:1789760786.108648:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27385856, bytes: 4096 00000080:00200000:0.0:1789760786.108655:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.108662:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.108664:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27385856, 27389952) 0 0 00000080:00200000:0.0:1789760786.108667:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.108681:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6686, 6686] 00020000:00200000:0.0:1789760786.108689:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27385856 chunk: [27385856, 27389952) 27389952, 4096 00020000:00200000:0.0:1789760786.108692:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.108707:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6686, 6686] 00000008:00010000:0.0:1789760786.108720:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.108725:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.108734:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27385856, 27389952) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.108747:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27385856=0x1a1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.108791:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.108793:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27389952 00000020:00010000:0.0:1789760786.108799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.108802:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.108805:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.108807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.108810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.108817:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.108823:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.108836:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760786.108927:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 27385856<>27389952 00000100:00100000:1.0:1789760786.109196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495939584:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.169300:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974003 gen 2 00000008:00010000:1.0:1789760786.169706:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760786.169713:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.169722:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.169743:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760786.169768:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495939584:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.169874:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6686, 6686] 00000080:00200000:0.0:1789760786.169886:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.169892:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.169903:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27389952 i_blocks 55305 00000020:00200000:0.0:1789760786.169907:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.169914:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.169941:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27389952, count: 4096 00000080:00200000:0.0:1789760786.169947:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27389952, bytes: 4096 00000080:00200000:0.0:1789760786.169955:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.169962:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.169964:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27389952, 27394048) 0 0 00000080:00200000:0.0:1789760786.169968:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.169980:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6687, 6687] 00020000:00200000:0.0:1789760786.169988:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27389952 chunk: [27389952, 27394048) 27394048, 4096 00020000:00200000:0.0:1789760786.169992:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.170007:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6687, 6687] 00000008:00010000:0.0:1789760786.170021:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.170026:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.170034:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27389952, 27394048) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.170046:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27389952=0x1a1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.170091:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.170094:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27394048 00000020:00010000:0.0:1789760786.170114:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.170118:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.170120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.170123:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.170125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.170132:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.170137:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.170151:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760786.170304:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 27389952<>27394048 00000100:00100000:1.0:1789760786.170873:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495939712:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.228970:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974004 gen 2 00000008:00010000:1.0:1789760786.229010:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760786.229017:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.229027:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.229049:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760786.229074:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495939712:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.229541:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6687, 6687] 00000080:00200000:0.0:1789760786.229555:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.229560:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.229571:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27394048 i_blocks 55305 00000020:00200000:0.0:1789760786.229574:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.229582:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.229671:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27394048, count: 4096 00000080:00200000:0.0:1789760786.229678:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27394048, bytes: 4096 00000080:00200000:0.0:1789760786.229687:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.229694:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.229696:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27394048, 27398144) 0 0 00000080:00200000:0.0:1789760786.229700:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.229712:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6688, 6688] 00020000:00200000:0.0:1789760786.229719:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27394048 chunk: [27394048, 27398144) 27398144, 4096 00020000:00200000:0.0:1789760786.229723:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.229738:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6688, 6688] 00000008:00010000:0.0:1789760786.229752:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.229757:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.229766:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27394048, 27398144) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.229778:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27394048=0x1a20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.229822:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.229825:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27398144 00000020:00010000:0.0:1789760786.229831:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.229835:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.229838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.229841:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.229844:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.229859:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.229865:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.229881:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760786.229972:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 27394048<>27398144 00000100:00100000:1.0:1789760786.230073:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495939840:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.276087:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974005 gen 2 00000008:00010000:1.0:1789760786.276128:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760786.276135:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.276143:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.276163:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760786.276213:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495939840:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.276521:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6688, 6688] 00000080:00200000:0.0:1789760786.276533:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.276539:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.276551:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27398144 i_blocks 55305 00000020:00200000:0.0:1789760786.276555:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.276563:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.276587:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27398144, count: 4096 00000080:00200000:0.0:1789760786.276593:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27398144, bytes: 4096 00000080:00200000:0.0:1789760786.276601:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.276608:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.276610:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27398144, 27402240) 0 0 00000080:00200000:0.0:1789760786.276626:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.276639:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6689, 6689] 00020000:00200000:0.0:1789760786.276645:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27398144 chunk: [27398144, 27402240) 27402240, 4096 00020000:00200000:0.0:1789760786.276649:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.276665:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6689, 6689] 00000008:00010000:0.0:1789760786.276678:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.276683:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.276692:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27398144, 27402240) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.276703:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27398144=0x1a21000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.276751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.276753:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27402240 00000020:00010000:0.0:1789760786.276760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.276763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.276766:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.276769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.276772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.276777:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.276783:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.276798:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760786.276904:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 27398144<>27402240 00000100:00100000:1.0:1789760786.277129:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495939968:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760786.323342:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760786.323352:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.323360:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.323374:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760786.323396:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495939968:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.323519:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6689, 6689] 00000080:00200000:0.0:1789760786.323532:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.323541:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.323552:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27402240 i_blocks 55305 00000020:00200000:0.0:1789760786.323555:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.323563:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.323588:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27402240, count: 4096 00000080:00200000:0.0:1789760786.323593:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27402240, bytes: 4096 00000080:00200000:0.0:1789760786.323609:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.323617:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.323710:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27402240, 27406336) 0 0 00000080:00200000:0.0:1789760786.323715:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.323727:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6690, 6690] 00020000:00200000:0.0:1789760786.323733:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27402240 chunk: [27402240, 27406336) 27406336, 4096 00020000:00200000:0.0:1789760786.323737:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.323752:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6690, 6690] 00000008:00010000:0.0:1789760786.323764:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.323769:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.323776:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27402240, 27406336) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.323787:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27402240=0x1a22000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.323828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.323830:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27406336 00000020:00010000:0.0:1789760786.323837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.323840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.323843:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.323846:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.323848:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.323854:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.323859:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.323872:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760786.323961:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 27402240<>27406336 00000100:00100000:1.0:1789760786.324046:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940096:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.376569:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974007 gen 2 00000008:00010000:1.0:1789760786.376628:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760786.376635:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.376643:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.376666:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760786.376690:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940096:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.376838:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6690, 6690] 00000080:00200000:0.0:1789760786.376852:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.376857:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.376868:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27406336 i_blocks 55305 00000020:00200000:0.0:1789760786.376871:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.376878:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.376902:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27406336, count: 4096 00000080:00200000:0.0:1789760786.376908:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27406336, bytes: 4096 00000080:00200000:0.0:1789760786.376915:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.376922:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.376923:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27406336, 27410432) 0 0 00000080:00200000:0.0:1789760786.376927:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.376939:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6691, 6691] 00020000:00200000:0.0:1789760786.376945:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27406336 chunk: [27406336, 27410432) 27410432, 4096 00020000:00200000:0.0:1789760786.376948:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.376963:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6691, 6691] 00000008:00010000:0.0:1789760786.376976:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.376981:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.376989:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27406336, 27410432) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.376999:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27406336=0x1a23000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.377044:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.377047:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27410432 00000020:00010000:0.0:1789760786.377054:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.377057:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.377060:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.377062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.377065:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.377070:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.377076:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.377090:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760786.377217:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 27406336<>27410432 00000100:00100000:1.0:1789760786.377468:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495940224:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.450906:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974008 gen 2 00000008:00010000:1.0:1789760786.450944:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760786.450951:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.450960:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.450980:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760786.451003:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495940224:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.451087:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6691, 6691] 00000080:00200000:0.0:1789760786.451098:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.451103:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.451115:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27410432 i_blocks 55305 00000020:00200000:0.0:1789760786.451118:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.451126:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.451152:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27410432, count: 4096 00000080:00200000:0.0:1789760786.451157:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27410432, bytes: 4096 00000080:00200000:0.0:1789760786.451164:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.451217:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.451219:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27410432, 27414528) 0 0 00000080:00200000:0.0:1789760786.451223:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.451254:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6692, 6692] 00020000:00200000:0.0:1789760786.451262:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27410432 chunk: [27410432, 27414528) 27414528, 4096 00020000:00200000:0.0:1789760786.451265:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.451291:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6692, 6692] 00000008:00010000:0.0:1789760786.451303:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.451309:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.451317:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27410432, 27414528) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.451328:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27410432=0x1a24000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.451368:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.451371:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27414528 00000020:00010000:0.0:1789760786.451377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.451380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.451383:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.451386:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.451389:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.451394:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.451400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.451412:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760786.451500:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 27410432<>27414528 00000100:00100000:1.0:1789760786.451946:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940352:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.514435:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974009 gen 2 00000008:00010000:1.0:1789760786.514467:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760786.514472:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.514480:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.514529:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760786.514550:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940352:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.514659:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6692, 6692] 00000080:00200000:0.0:1789760786.514671:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.514676:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.514687:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27414528 i_blocks 55305 00000020:00200000:0.0:1789760786.514690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.514697:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.514719:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27414528, count: 4096 00000080:00200000:0.0:1789760786.514724:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27414528, bytes: 4096 00000080:00200000:0.0:1789760786.514730:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.514738:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.514739:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27414528, 27418624) 0 0 00000080:00200000:0.0:1789760786.514743:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.514755:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6693, 6693] 00020000:00200000:0.0:1789760786.514763:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27414528 chunk: [27414528, 27418624) 27418624, 4096 00020000:00200000:0.0:1789760786.514766:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.514901:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6693, 6693] 00000008:00010000:0.0:1789760786.514981:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.514987:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.515064:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27414528, 27418624) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.515076:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27414528=0x1a25000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.515123:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.515125:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27418624 00000020:00010000:0.0:1789760786.515132:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.515135:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.515138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.515141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.515338:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.515345:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.515351:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.515368:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760786.515466:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 27414528<>27418624 00000100:00100000:1.0:1789760786.515523:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495940480:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.561397:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974010 gen 2 00000008:00010000:1.0:1789760786.561435:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760786.561442:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.561450:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.561469:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760786.561494:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495940480:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.561820:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6693, 6693] 00000080:00200000:0.0:1789760786.561833:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.561838:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.561850:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27418624 i_blocks 55305 00000020:00200000:0.0:1789760786.561853:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.561860:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.561888:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27418624, count: 4096 00000080:00200000:0.0:1789760786.561894:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27418624, bytes: 4096 00000080:00200000:0.0:1789760786.561901:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.561908:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.561910:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27418624, 27422720) 0 0 00000080:00200000:0.0:1789760786.561914:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.561926:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6694, 6694] 00020000:00200000:0.0:1789760786.561933:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27418624 chunk: [27418624, 27422720) 27422720, 4096 00020000:00200000:0.0:1789760786.561938:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.561954:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6694, 6694] 00000008:00010000:0.0:1789760786.561967:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.561973:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.561981:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27418624, 27422720) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.562144:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27418624=0x1a26000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.562286:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.562289:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27422720 00000020:00010000:0.0:1789760786.562297:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.562300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.562303:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.562306:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.562309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.562315:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.562320:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.562335:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760786.562429:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 27418624<>27422720 00000100:00100000:1.0:1789760786.562764:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940608:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.631004:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974011 gen 2 00000008:00010000:1.0:1789760786.631055:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760786.631064:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.631076:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.631097:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760786.631126:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940608:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.632242:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6694, 6694] 00000080:00200000:0.0:1789760786.632262:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.632279:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.632298:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27422720 i_blocks 55305 00000020:00200000:0.0:1789760786.632302:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.632313:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.632349:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27422720, count: 4096 00000080:00200000:0.0:1789760786.632357:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27422720, bytes: 4096 00000080:00200000:0.0:1789760786.632368:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.632377:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.632380:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27422720, 27426816) 0 0 00000080:00200000:0.0:1789760786.632385:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.632402:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6695, 6695] 00020000:00200000:0.0:1789760786.632410:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27422720 chunk: [27422720, 27426816) 27426816, 4096 00020000:00200000:0.0:1789760786.632415:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.632435:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6695, 6695] 00000008:00010000:0.0:1789760786.632452:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.632458:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.632469:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27422720, 27426816) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.632486:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27422720=0x1a27000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.632562:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.632565:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27426816 00000020:00010000:0.0:1789760786.632574:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.632577:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.632580:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.632584:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.632587:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.634946:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.634966:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.635003:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760786.635147:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 27422720<>27426816 00000100:00100000:1.0:1789760786.636207:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760786.636241:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940736:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.714237:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974012 gen 2 00000008:00010000:1.0:1789760786.714295:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760786.714304:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.714316:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.714339:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760786.714367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940736:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.714533:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6695, 6695] 00000080:00200000:0.0:1789760786.714549:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.714556:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.714574:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27426816 i_blocks 55305 00000020:00200000:0.0:1789760786.714578:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.714587:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.714636:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27426816, count: 4096 00000080:00200000:0.0:1789760786.714643:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27426816, bytes: 4096 00000080:00200000:0.0:1789760786.714654:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.714664:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.714666:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27426816, 27430912) 0 0 00000080:00200000:0.0:1789760786.714671:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.714688:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6696, 6696] 00020000:00200000:0.0:1789760786.714696:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27426816 chunk: [27426816, 27430912) 27430912, 4096 00020000:00200000:0.0:1789760786.714702:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.714721:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6696, 6696] 00000008:00010000:0.0:1789760786.714739:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.714745:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.714756:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27426816, 27430912) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.714769:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27426816=0x1a28000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.714823:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.714826:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27430912 00000020:00010000:0.0:1789760786.714834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.714837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.714840:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.714844:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.714847:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.714854:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.714861:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.714878:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760786.716382:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 27426816<>27430912 00000100:00100000:1.0:1789760786.719141:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940864:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.753642:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974013 gen 2 00000008:00010000:1.0:1789760786.753693:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760786.753701:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.753711:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.753794:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760786.753828:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940864:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.755792:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6696, 6696] 00000080:00200000:0.0:1789760786.755820:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.755888:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.755968:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27430912 i_blocks 55305 00000020:00200000:0.0:1789760786.755975:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.755991:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.756027:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27430912, count: 4096 00000080:00200000:0.0:1789760786.756036:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27430912, bytes: 4096 00000080:00200000:0.0:1789760786.756051:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.756062:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.756064:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27430912, 27435008) 0 0 00000080:00200000:0.0:1789760786.756068:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.756087:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6697, 6697] 00020000:00200000:0.0:1789760786.756096:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27430912 chunk: [27430912, 27435008) 27435008, 4096 00020000:00200000:0.0:1789760786.756102:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.756125:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6697, 6697] 00000008:00010000:0.0:1789760786.756146:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.756152:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.756165:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27430912, 27435008) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.758075:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27430912=0x1a29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.760400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.760414:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27435008 00000020:00010000:0.0:1789760786.760432:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.760437:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.760442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.760446:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.760450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.760465:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.760476:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.760515:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760786.761140:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 27430912<>27435008 00000100:00100000:1.0:1789760786.761665:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760786.761695:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940992:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.801449:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974014 gen 2 00000008:00010000:1.0:1789760786.801489:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760786.801495:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.801503:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.801521:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760786.801543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495940992:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760786.801680:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6697, 6697] 00000080:00200000:0.0:1789760786.801693:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.801698:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.801711:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27435008 i_blocks 55305 00000020:00200000:0.0:1789760786.801714:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.801722:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.801826:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27435008, count: 4096 00000080:00200000:0.0:1789760786.801833:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27435008, bytes: 4096 00000080:00200000:0.0:1789760786.801841:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.801927:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.801930:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27435008, 27439104) 0 0 00000080:00200000:0.0:1789760786.801934:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.801948:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6698, 6698] 00020000:00200000:0.0:1789760786.801957:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27435008 chunk: [27435008, 27439104) 27439104, 4096 00020000:00200000:0.0:1789760786.801962:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.801977:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6698, 6698] 00000008:00010000:0.0:1789760786.802058:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.802064:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760786.802073:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27435008, 27439104) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.802086:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27435008=0x1a2a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.802135:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.802138:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27439104 00000020:00010000:0.0:1789760786.802146:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.802149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.802152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.802155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.802158:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.802164:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.802195:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.802210:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760786.802575:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 27435008<>27439104 00000100:00100000:0.0:1789760786.802842:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760786.802969:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495941120:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760786.875732:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974015 gen 2 00000008:00010000:1.0:1789760786.875768:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760786.875774:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.875783:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.875807:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760786.875833:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495941120:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760786.876035:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6698, 6698] 00000080:00200000:3.0:1789760786.876047:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.876052:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.876064:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27439104 i_blocks 55305 00000020:00200000:3.0:1789760786.876067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.876075:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.876099:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27439104, count: 4096 00000080:00200000:3.0:1789760786.876105:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27439104, bytes: 4096 00000080:00200000:3.0:1789760786.876115:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.876122:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.876124:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27439104, 27443200) 0 0 00000080:00200000:3.0:1789760786.876128:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.876140:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6699, 6699] 00020000:00200000:3.0:1789760786.876147:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27439104 chunk: [27439104, 27443200) 27443200, 4096 00020000:00200000:3.0:1789760786.876151:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.876165:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6699, 6699] 00000008:00010000:3.0:1789760786.876215:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.876220:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203aceb0, osc lock ffff8dd205053b50, flags 0x0 00000080:00200000:3.0:1789760786.876229:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27439104, 27443200) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.876569:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27439104=0x1a2b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.876616:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.876619:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27443200 00000020:00010000:2.0:1789760786.876626:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.876629:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.876632:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203aceb0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.876635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.876638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203aceb0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.876645:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.876651:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.876663:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760786.876757:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 27439104<>27443200 00000100:00100000:3.0:1789760786.876985:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.877001:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495941248:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760786.891780:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760786.919387:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974016 gen 2 00000008:00010000:3.0:1789760786.919424:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760786.919431:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.919440:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.919462:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760786.919485:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495941248:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760786.920015:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6699, 6699] 00000080:00200000:2.0:1789760786.920028:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.920034:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.920046:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27443200 i_blocks 55305 00000020:00200000:2.0:1789760786.920050:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.920059:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.920082:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27443200, count: 4096 00000080:00200000:2.0:1789760786.920088:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27443200, bytes: 4096 00000080:00200000:2.0:1789760786.920097:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.920103:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.920105:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27443200, 27447296) 0 0 00000080:00200000:2.0:1789760786.920109:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.920119:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6700, 6700] 00020000:00200000:2.0:1789760786.920126:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27443200 chunk: [27443200, 27447296) 27447296, 4096 00020000:00200000:2.0:1789760786.920130:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.920143:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6700, 6700] 00000008:00010000:2.0:1789760786.920156:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.920161:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760786.920189:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27443200, 27447296) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.920200:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27443200=0x1a2c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.920250:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.920253:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27447296 00000020:00010000:2.0:1789760786.920260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.920264:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.920276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.920279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.920282:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.920289:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.920295:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.920315:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760786.920408:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 27443200<>27447296 00000100:00100000:3.0:1789760786.921013:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760786.921031:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495941376:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760787.004416:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974017 gen 2 00000008:00010000:3.0:1789760787.004473:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760787.004481:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.004493:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.004579:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760787.004610:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495941376:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.004708:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6700, 6700] 00000080:00200000:2.0:1789760787.004725:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.004732:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.004747:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27447296 i_blocks 55305 00000020:00200000:2.0:1789760787.004751:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.004761:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.004791:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27447296, count: 4096 00000080:00200000:2.0:1789760787.004798:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27447296, bytes: 4096 00000080:00200000:2.0:1789760787.004807:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.004816:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.004819:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27447296, 27451392) 0 0 00000080:00200000:2.0:1789760787.004824:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.004844:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6701, 6701] 00020000:00200000:2.0:1789760787.004907:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27447296 chunk: [27447296, 27451392) 27451392, 4096 00020000:00200000:2.0:1789760787.004913:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.004934:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6701, 6701] 00000008:00010000:2.0:1789760787.004952:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.004958:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.004970:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27447296, 27451392) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.004985:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27447296=0x1a2d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.005049:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.005052:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27451392 00000020:00010000:2.0:1789760787.005062:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.005066:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.005069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.005072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.005075:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.005083:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.005090:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.005107:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760787.005441:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 27447296<>27451392 00000100:00100000:2.0:1789760787.005480:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760787.005499:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495941504:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760787.032496:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974018 gen 2 00000008:00010000:2.0:1789760787.032545:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760787.032553:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.032563:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.032594:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760787.032622:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495941504:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760787.032883:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6701, 6701] 00000080:00200000:3.0:1789760787.032900:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.032907:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.032923:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27451392 i_blocks 55305 00000020:00200000:3.0:1789760787.032927:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.032937:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.032974:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27451392, count: 4096 00000080:00200000:3.0:1789760787.032981:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27451392, bytes: 4096 00000080:00200000:3.0:1789760787.032991:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.033001:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.033004:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27451392, 27455488) 0 0 00000080:00200000:3.0:1789760787.033009:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.033025:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6702, 6702] 00020000:00200000:3.0:1789760787.033034:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27451392 chunk: [27451392, 27455488) 27455488, 4096 00020000:00200000:3.0:1789760787.033039:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.033058:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6702, 6702] 00000008:00010000:3.0:1789760787.033080:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.033086:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760787.033098:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27451392, 27455488) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.033114:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27451392=0x1a2e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.033337:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.033341:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27455488 00000020:00010000:3.0:1789760787.033350:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.033354:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.033358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.033361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.033364:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.033372:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.033380:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.033397:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760787.033521:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 27451392<>27455488 00000100:00100000:2.0:1789760787.033960:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495941632:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760787.062094:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:2.0:1789760787.062111:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.062123:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.062151:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:2.0:1789760787.062204:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495941632:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760787.062299:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6702, 6702] 00000080:00200000:1.0:1789760787.062315:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.062321:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.062337:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27455488 i_blocks 55305 00000020:00200000:1.0:1789760787.062341:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.062351:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.062411:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27455488, count: 4096 00000080:00200000:1.0:1789760787.062419:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27455488, bytes: 4096 00000080:00200000:1.0:1789760787.062430:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.062447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760787.062450:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27455488, 27459584) 0 0 00000080:00200000:1.0:1789760787.062454:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.062835:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6703, 6703] 00020000:00200000:1.0:1789760787.062845:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27455488 chunk: [27455488, 27459584) 27459584, 4096 00020000:00200000:1.0:1789760787.062850:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.062875:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6703, 6703] 00000008:00010000:1.0:1789760787.062895:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.062902:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760787.062913:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27455488, 27459584) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.062929:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27455488=0x1a2f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.062992:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.062995:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27459584 00000020:00010000:1.0:1789760787.063004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.063007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.063010:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.063013:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.063016:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.063023:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.063030:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.063048:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760787.063207:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27455488<>27459584 00000100:00100000:1.0:1789760787.063245:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760787.063262:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495941760:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.098030:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974020 gen 2 00000008:00010000:1.0:1789760787.098097:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760787.098106:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.098117:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.098139:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760787.098225:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495941760:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760787.098870:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6703, 6703] 00000080:00200000:0.0:1789760787.098888:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.098895:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.098911:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27459584 i_blocks 55305 00000020:00200000:0.0:1789760787.098916:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.098926:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.098962:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27459584, count: 4096 00000080:00200000:0.0:1789760787.098969:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27459584, bytes: 4096 00000080:00200000:0.0:1789760787.098980:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.098989:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.098991:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27459584, 27463680) 0 0 00000080:00200000:0.0:1789760787.098996:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.099012:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6704, 6704] 00020000:00200000:0.0:1789760787.099020:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27459584 chunk: [27459584, 27463680) 27463680, 4096 00020000:00200000:0.0:1789760787.099025:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.099048:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6704, 6704] 00000008:00010000:0.0:1789760787.099065:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.099071:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760787.099082:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27459584, 27463680) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.099096:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27459584=0x1a30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.099157:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.099160:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27463680 00000020:00010000:0.0:1789760787.099192:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.099196:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.099199:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.099203:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.099205:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.099214:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.099220:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.099238:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760787.099543:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 27459584<>27463680 00000100:00100000:0.0:1789760787.099592:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760787.099619:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495941888:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.145565:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974021 gen 2 00000008:00010000:1.0:1789760787.145615:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760787.145624:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.145636:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.145655:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760787.145684:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495941888:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760787.145788:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6704, 6704] 00000080:00200000:0.0:1789760787.145858:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.145865:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.145881:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27463680 i_blocks 55305 00000020:00200000:0.0:1789760787.145885:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.145894:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.145928:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27463680, count: 4096 00000080:00200000:0.0:1789760787.145935:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27463680, bytes: 4096 00000080:00200000:0.0:1789760787.145944:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.145954:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.145956:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27463680, 27467776) 0 0 00000080:00200000:0.0:1789760787.145961:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.145977:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6705, 6705] 00020000:00200000:0.0:1789760787.145985:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27463680 chunk: [27463680, 27467776) 27467776, 4096 00020000:00200000:0.0:1789760787.145990:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.146009:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6705, 6705] 00000008:00010000:0.0:1789760787.146027:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.146033:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760787.146044:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27463680, 27467776) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.146057:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27463680=0x1a31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.146114:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.146117:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27467776 00000020:00010000:0.0:1789760787.146125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.146128:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.146131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.146135:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.146138:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.146145:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.146152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.147148:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760787.147629:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 27463680<>27467776 00000100:00100000:1.0:1789760787.147744:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495942016:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.181657:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974022 gen 2 00000008:00010000:1.0:1789760787.181710:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760787.181719:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.182096:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.182215:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760787.182245:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495942016:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760787.184328:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6705, 6705] 00000080:00200000:3.0:1789760787.184355:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.184361:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.184378:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27467776 i_blocks 55305 00000020:00200000:3.0:1789760787.184383:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.184398:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.184432:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27467776, count: 4096 00000080:00200000:3.0:1789760787.184441:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27467776, bytes: 4096 00000080:00200000:3.0:1789760787.184456:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.184466:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.184468:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27467776, 27471872) 0 0 00000080:00200000:3.0:1789760787.184473:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.184491:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6706, 6706] 00020000:00200000:3.0:1789760787.184499:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27467776 chunk: [27467776, 27471872) 27471872, 4096 00020000:00200000:3.0:1789760787.184504:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.184526:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6706, 6706] 00000008:00010000:3.0:1789760787.184543:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.184548:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760787.184559:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27467776, 27471872) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.184572:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27467776=0x1a32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.184639:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.184642:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27471872 00000020:00010000:3.0:1789760787.184651:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.184654:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.184658:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.184661:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.184664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.184672:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.184679:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.184721:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760787.184867:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 27467776<>27471872 00000100:00100000:2.0:1789760787.185232:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495942144:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760787.232420:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760787.232435:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.232448:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.232469:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760787.232502:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495942144:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760787.232504:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6706, 6706] 00000080:00200000:3.0:1789760787.232524:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.232531:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.232547:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27471872 i_blocks 55305 00000020:00200000:3.0:1789760787.232550:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.232562:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.233204:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27471872, count: 4096 00000080:00200000:2.0:1789760787.233214:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27471872, bytes: 4096 00000080:00200000:2.0:1789760787.233228:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.233237:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.233239:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27471872, 27475968) 0 0 00000080:00200000:2.0:1789760787.233244:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.233261:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6707, 6707] 00020000:00200000:2.0:1789760787.233282:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27471872 chunk: [27471872, 27475968) 27475968, 4096 00020000:00200000:2.0:1789760787.233287:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.233344:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6707, 6707] 00000008:00010000:2.0:1789760787.233362:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.233367:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.233379:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27471872, 27475968) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.233394:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27471872=0x1a33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.233457:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.233459:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27475968 00000020:00010000:2.0:1789760787.233468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.233472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.233475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.233479:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.233482:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.233492:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.233498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.233536:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760787.233660:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 27471872<>27475968 00000100:00100000:2.0:1789760787.233710:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760787.233729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495942272:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760787.316871:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974023 gen 2 00000100:00100000:2.0:1789760787.316914:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264380 x1876699495942272/t4294974024(4294974024) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760787.316964:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760787.316972:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.316983:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.317012:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760787.317035:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495942272:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760787.318831:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6707, 6707] 00000080:00200000:1.0:1789760787.318861:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.318868:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.318885:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27475968 i_blocks 55305 00000020:00200000:1.0:1789760787.318890:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.318904:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.318938:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27475968, count: 4096 00000080:00200000:1.0:1789760787.318947:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27475968, bytes: 4096 00000080:00200000:1.0:1789760787.318963:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.318973:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760787.318975:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27475968, 27480064) 0 0 00000080:00200000:1.0:1789760787.318980:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.318998:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6708, 6708] 00020000:00200000:1.0:1789760787.319007:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27475968 chunk: [27475968, 27480064) 27480064, 4096 00020000:00200000:1.0:1789760787.319012:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.319033:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6708, 6708] 00000008:00010000:1.0:1789760787.319053:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.319059:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760787.319072:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27475968, 27480064) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.319087:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27475968=0x1a34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.319152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.319156:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27480064 00000020:00010000:1.0:1789760787.319164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.319220:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.319224:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.319228:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.319231:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.319240:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.319248:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.319280:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760787.319410:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27475968<>27480064 00000100:00100000:1.0:1789760787.319460:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760787.319484:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495942400:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.380608:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974025 gen 2 00000008:00010000:1.0:1789760787.380673:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760787.380681:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.380691:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.380717:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760787.380745:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495942400:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760787.380770:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6708, 6708] 00000080:00200000:0.0:1789760787.380789:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.380795:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.380812:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27480064 i_blocks 55305 00000020:00200000:0.0:1789760787.380816:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.380826:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.380870:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27480064, count: 4096 00000080:00200000:0.0:1789760787.380878:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27480064, bytes: 4096 00000080:00200000:0.0:1789760787.380891:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.380901:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.380903:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27480064, 27484160) 0 0 00000080:00200000:0.0:1789760787.380908:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.380927:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6709, 6709] 00020000:00200000:0.0:1789760787.380935:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27480064 chunk: [27480064, 27484160) 27484160, 4096 00020000:00200000:0.0:1789760787.380940:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.380961:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6709, 6709] 00000008:00010000:0.0:1789760787.380979:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.380985:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760787.380995:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27480064, 27484160) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.381010:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27480064=0x1a35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.381067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.381070:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27484160 00000020:00010000:0.0:1789760787.381080:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.381083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.381086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.381090:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.381093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.381100:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.381108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.381123:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760787.381303:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 27480064<>27484160 00000100:00100000:1.0:1789760787.381394:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495942528:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.424774:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974026 gen 2 00000008:00010000:1.0:1789760787.424834:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760787.424843:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.424853:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.424884:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760787.424914:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495942528:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.425087:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6709, 6709] 00000080:00200000:2.0:1789760787.425106:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.425112:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.425129:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27484160 i_blocks 55305 00000020:00200000:2.0:1789760787.425133:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.425143:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.425245:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27484160, count: 4096 00000080:00200000:2.0:1789760787.425253:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27484160, bytes: 4096 00000080:00200000:2.0:1789760787.425272:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.425281:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.425283:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27484160, 27488256) 0 0 00000080:00200000:2.0:1789760787.425287:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.425305:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6710, 6710] 00020000:00200000:2.0:1789760787.425313:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27484160 chunk: [27484160, 27488256) 27488256, 4096 00020000:00200000:2.0:1789760787.425318:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.425338:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6710, 6710] 00000008:00010000:2.0:1789760787.425356:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.425362:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.425373:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27484160, 27488256) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.425387:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27484160=0x1a36000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.425444:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.425447:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27488256 00000020:00010000:2.0:1789760787.425456:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.425464:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.425467:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.425471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.425475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.425483:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.425490:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.425506:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760787.425696:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 27484160<>27488256 00000100:00100000:3.0:1789760787.425814:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495942656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760787.466920:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974027 gen 2 00000008:00010000:3.0:1789760787.466978:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760787.466987:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.466997:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.467026:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760787.467057:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495942656:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.467958:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6710, 6710] 00000080:00200000:2.0:1789760787.467981:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.467988:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.468005:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27488256 i_blocks 55305 00000020:00200000:2.0:1789760787.468009:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.468021:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.468056:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27488256, count: 4096 00000080:00200000:2.0:1789760787.468064:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27488256, bytes: 4096 00000080:00200000:2.0:1789760787.468077:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.468087:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.468089:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27488256, 27492352) 0 0 00000080:00200000:2.0:1789760787.468093:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.468110:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6711, 6711] 00020000:00200000:2.0:1789760787.468119:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27488256 chunk: [27488256, 27492352) 27492352, 4096 00020000:00200000:2.0:1789760787.468124:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.468146:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6711, 6711] 00000008:00010000:2.0:1789760787.468164:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.468906:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.468929:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27488256, 27492352) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.468947:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27488256=0x1a37000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.469016:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.469019:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27492352 00000020:00010000:2.0:1789760787.469029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.469033:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.469036:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.469039:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.469043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.469052:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.469060:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.469087:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760787.469910:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 27488256<>27492352 00000100:00100000:3.0:1789760787.469988:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495942784:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760787.538893:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760787.538909:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.538921:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.538952:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760787.538984:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495942784:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760787.539087:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6711, 6711] 00000080:00200000:1.0:1789760787.539105:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.539111:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.539127:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27492352 i_blocks 55305 00000020:00200000:1.0:1789760787.539130:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.539141:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.539202:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27492352, count: 4096 00000080:00200000:1.0:1789760787.539210:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27492352, bytes: 4096 00000080:00200000:1.0:1789760787.539221:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.539230:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760787.539232:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27492352, 27496448) 0 0 00000080:00200000:1.0:1789760787.539237:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.539272:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6712, 6712] 00020000:00200000:1.0:1789760787.539280:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27492352 chunk: [27492352, 27496448) 27496448, 4096 00020000:00200000:1.0:1789760787.539284:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.539304:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6712, 6712] 00000008:00010000:1.0:1789760787.539320:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.539325:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760787.539336:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27492352, 27496448) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.539350:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27492352=0x1a38000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.539407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.539411:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27496448 00000020:00010000:1.0:1789760787.539420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.539423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.539426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.539430:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.539433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.539441:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.539448:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.539465:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760787.539780:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27492352<>27496448 00000100:00100000:1.0:1789760787.539836:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760787.539855:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495942912:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.577955:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974029 gen 2 00000008:00010000:1.0:1789760787.578093:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760787.578102:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.578113:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.578210:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760787.578238:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495942912:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760787.578407:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6712, 6712] 00000080:00200000:0.0:1789760787.578423:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.578429:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.578444:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27496448 i_blocks 55305 00000020:00200000:0.0:1789760787.578448:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.578457:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.578491:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27496448, count: 4096 00000080:00200000:0.0:1789760787.578497:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27496448, bytes: 4096 00000080:00200000:0.0:1789760787.578508:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.578517:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.578519:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27496448, 27500544) 0 0 00000080:00200000:0.0:1789760787.578524:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.578541:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6713, 6713] 00020000:00200000:0.0:1789760787.578548:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27496448 chunk: [27496448, 27500544) 27500544, 4096 00020000:00200000:0.0:1789760787.578552:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.578570:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6713, 6713] 00000008:00010000:0.0:1789760787.578588:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.578593:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760787.578604:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27496448, 27500544) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.578619:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27496448=0x1a39000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.578676:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.578679:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27500544 00000020:00010000:0.0:1789760787.578688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.578691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.578694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.578697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.578700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.578707:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.578714:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.578730:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760787.578878:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 27496448<>27500544 00000100:00100000:1.0:1789760787.578940:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495943040:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.630436:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974030 gen 2 00000008:00010000:1.0:1789760787.630490:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760787.630498:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.630509:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.630532:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760787.630561:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495943040:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760787.630734:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6713, 6713] 00000080:00200000:0.0:1789760787.630752:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.630758:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.630775:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27500544 i_blocks 55305 00000020:00200000:0.0:1789760787.630779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.630788:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.630822:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27500544, count: 4096 00000080:00200000:0.0:1789760787.630830:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27500544, bytes: 4096 00000080:00200000:0.0:1789760787.630840:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.630929:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.630933:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27500544, 27504640) 0 0 00000080:00200000:0.0:1789760787.630938:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.630957:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6714, 6714] 00020000:00200000:0.0:1789760787.630967:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27500544 chunk: [27500544, 27504640) 27504640, 4096 00020000:00200000:0.0:1789760787.630972:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.630992:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6714, 6714] 00000008:00010000:0.0:1789760787.631012:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.631018:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760787.631029:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27500544, 27504640) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.631043:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27500544=0x1a3a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.631104:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.631107:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27504640 00000020:00010000:0.0:1789760787.631117:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.631120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.631123:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.631127:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.631130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.631137:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.631144:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.631162:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760787.631936:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 27500544<>27504640 00000100:00100000:1.0:1789760787.632008:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760787.632098:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495943168:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.662521:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974031 gen 2 00000008:00010000:1.0:1789760787.662572:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760787.662581:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.662603:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.662627:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760787.662658:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495943168:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760787.662817:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6714, 6714] 00000080:00200000:3.0:1789760787.662864:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.662871:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.662887:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27504640 i_blocks 55305 00000020:00200000:3.0:1789760787.662891:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.662900:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.662947:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27504640, count: 4096 00000080:00200000:3.0:1789760787.662956:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27504640, bytes: 4096 00000080:00200000:3.0:1789760787.662967:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.662976:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.662978:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27504640, 27508736) 0 0 00000080:00200000:3.0:1789760787.662983:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.662998:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6715, 6715] 00020000:00200000:3.0:1789760787.663006:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27504640 chunk: [27504640, 27508736) 27508736, 4096 00020000:00200000:3.0:1789760787.663011:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.663032:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6715, 6715] 00000008:00010000:3.0:1789760787.663050:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.663056:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760787.663067:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27504640, 27508736) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.663081:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27504640=0x1a3b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.663135:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.663139:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27508736 00000020:00010000:3.0:1789760787.663148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.663151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.663155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.663158:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.663161:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.663213:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.663221:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.663237:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760787.663366:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 27504640<>27508736 00000100:00100000:3.0:1789760787.663416:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760787.663433:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495943296:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760787.699947:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:3.0:1789760787.699963:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.699974:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.700000:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:3.0:1789760787.700030:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495943296:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.700136:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6715, 6715] 00000080:00200000:2.0:1789760787.700153:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.700160:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.700206:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27508736 i_blocks 55305 00000020:00200000:2.0:1789760787.700211:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.700222:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.700257:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27508736, count: 4096 00000080:00200000:2.0:1789760787.700274:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27508736, bytes: 4096 00000080:00200000:2.0:1789760787.700285:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.700295:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.700298:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27508736, 27512832) 0 0 00000080:00200000:2.0:1789760787.700302:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.700319:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6716, 6716] 00020000:00200000:2.0:1789760787.700328:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27508736 chunk: [27508736, 27512832) 27512832, 4096 00020000:00200000:2.0:1789760787.700333:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.700354:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6716, 6716] 00000008:00010000:2.0:1789760787.700374:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.700380:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.700391:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27508736, 27512832) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.700407:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27508736=0x1a3c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.700469:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.700472:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27512832 00000020:00010000:2.0:1789760787.700481:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.700484:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.700487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.700491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.700495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.700504:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.700512:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.700529:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760787.700669:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 27508736<>27512832 00000100:00100000:2.0:1789760787.700713:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760787.700729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495943424:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760787.735010:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974032 gen 2 00000100:00100000:2.0:1789760787.735054:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205264e00 x1876699495943424/t4294974033(4294974033) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760787.735104:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:2.0:1789760787.735112:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.735122:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.735150:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:2.0:1789760787.735215:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495943424:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760787.735418:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6716, 6716] 00000080:00200000:1.0:1789760787.735437:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.735444:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.735460:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27512832 i_blocks 55305 00000020:00200000:1.0:1789760787.735464:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.735474:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.735512:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27512832, count: 4096 00000080:00200000:1.0:1789760787.735519:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27512832, bytes: 4096 00000080:00200000:1.0:1789760787.735531:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.735540:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760787.735542:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27512832, 27516928) 0 0 00000080:00200000:1.0:1789760787.735547:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.735565:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6717, 6717] 00020000:00200000:1.0:1789760787.735573:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27512832 chunk: [27512832, 27516928) 27516928, 4096 00020000:00200000:1.0:1789760787.735577:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.735609:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6717, 6717] 00000008:00010000:1.0:1789760787.735628:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.735634:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760787.735646:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27512832, 27516928) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.735660:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27512832=0x1a3d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.735716:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.735718:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27516928 00000020:00010000:1.0:1789760787.735726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.735730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.735733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.735737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.735740:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.735747:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.735754:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.735771:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760787.735968:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27512832<>27516928 00000100:00100000:0.0:1789760787.736043:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495943552:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760787.776571:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974034 gen 2 00000008:00010000:1.0:1789760787.776696:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760787.776704:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.776715:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.776745:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760787.776774:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495943552:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.776892:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6717, 6717] 00000080:00200000:2.0:1789760787.776909:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.776916:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.776931:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27516928 i_blocks 55305 00000020:00200000:2.0:1789760787.776935:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.776944:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.776979:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27516928, count: 4096 00000080:00200000:2.0:1789760787.776986:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27516928, bytes: 4096 00000080:00200000:2.0:1789760787.776997:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.777007:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.777009:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27516928, 27521024) 0 0 00000080:00200000:2.0:1789760787.777014:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.777128:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6718, 6718] 00020000:00200000:2.0:1789760787.777138:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27516928 chunk: [27516928, 27521024) 27521024, 4096 00020000:00200000:2.0:1789760787.777143:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.777166:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6718, 6718] 00000008:00010000:2.0:1789760787.777204:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.777211:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.777223:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27516928, 27521024) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.777238:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27516928=0x1a3e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.777306:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.777309:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27521024 00000020:00010000:2.0:1789760787.777317:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.777320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.777323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.777327:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.777330:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.777337:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.777344:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.777360:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760787.777493:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 27516928<>27521024 00000100:00100000:3.0:1789760787.777569:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495943680:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760787.812817:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760787.812833:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.812845:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.812965:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760787.812999:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495943680:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.813513:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6718, 6718] 00000080:00200000:2.0:1789760787.813532:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.813538:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.813554:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27521024 i_blocks 55305 00000020:00200000:2.0:1789760787.813559:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.813569:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.813613:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27521024, count: 4096 00000080:00200000:2.0:1789760787.813621:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27521024, bytes: 4096 00000080:00200000:2.0:1789760787.813636:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.813645:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.813647:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27521024, 27525120) 0 0 00000080:00200000:2.0:1789760787.813652:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.813669:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6719, 6719] 00020000:00200000:2.0:1789760787.813678:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27521024 chunk: [27521024, 27525120) 27525120, 4096 00020000:00200000:2.0:1789760787.813683:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.813704:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6719, 6719] 00000008:00010000:2.0:1789760787.813723:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.813729:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.813741:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27521024, 27525120) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.813755:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27521024=0x1a3f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.813814:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.813817:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27525120 00000020:00010000:2.0:1789760787.813826:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.813829:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.813948:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.813953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.813956:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.813964:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.813972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.813991:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760787.814115:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 27521024<>27525120 00000100:00100000:3.0:1789760787.814304:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495943808:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760787.857748:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974035 gen 2 00000100:00100000:3.0:1789760787.857796:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd205267b80 x1876699495943808/t4294974036(4294974036) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760787.858475:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760787.858485:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.858495:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.858520:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760787.858547:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495943808:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.858702:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6719, 6719] 00000080:00200000:2.0:1789760787.858720:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.858726:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.858743:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27525120 i_blocks 55305 00000020:00200000:2.0:1789760787.858747:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.858757:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.858791:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27525120, count: 4096 00000080:00200000:2.0:1789760787.858799:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27525120, bytes: 4096 00000080:00200000:2.0:1789760787.858811:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.858819:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.858822:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27525120, 27529216) 0 0 00000080:00200000:2.0:1789760787.858826:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.858855:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6720, 6720] 00020000:00200000:2.0:1789760787.858863:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27525120 chunk: [27525120, 27529216) 27529216, 4096 00020000:00200000:2.0:1789760787.858868:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.858887:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6720, 6720] 00000008:00010000:2.0:1789760787.858905:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.858910:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.859455:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27525120, 27529216) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.859474:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27525120=0x1a40000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.859539:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.859542:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27529216 00000020:00010000:2.0:1789760787.859551:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.859554:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.859558:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.859561:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.859564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.859574:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.859581:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.859604:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760787.859737:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 27525120<>27529216 00000100:00100000:3.0:1789760787.860615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495943936:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760787.894919:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974037 gen 2 00000008:00010000:3.0:1789760787.894982:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760787.894990:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.895003:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.895032:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760787.895063:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495943936:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.895206:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6720, 6720] 00000080:00200000:2.0:1789760787.895224:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.895231:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.895248:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27529216 i_blocks 55305 00000020:00200000:2.0:1789760787.895252:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.895270:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.895302:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27529216, count: 4096 00000080:00200000:2.0:1789760787.895310:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27529216, bytes: 4096 00000080:00200000:2.0:1789760787.895320:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.895329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.895331:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27529216, 27533312) 0 0 00000080:00200000:2.0:1789760787.895336:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.895352:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6721, 6721] 00020000:00200000:2.0:1789760787.895360:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27529216 chunk: [27529216, 27533312) 27533312, 4096 00020000:00200000:2.0:1789760787.895375:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.895430:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6721, 6721] 00000008:00010000:2.0:1789760787.895450:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.895456:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.895468:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27529216, 27533312) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.895483:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27529216=0x1a41000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.895541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.895543:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27533312 00000020:00010000:2.0:1789760787.895552:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.895555:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.895558:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.895562:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.895564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.895571:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.895578:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.895598:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760787.895721:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 27529216<>27533312 00000100:00100000:3.0:1789760787.895885:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495944064:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760787.916447:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760787.946470:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974038 gen 2 00000008:00010000:3.0:1789760787.946523:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760787.946532:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.946545:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.946570:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760787.946601:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495944064:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760787.946656:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6721, 6721] 00000080:00200000:2.0:1789760787.946672:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.946677:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.946690:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27533312 i_blocks 55305 00000020:00200000:2.0:1789760787.946693:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.946701:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.946777:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27533312, count: 4096 00000080:00200000:2.0:1789760787.946783:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27533312, bytes: 4096 00000080:00200000:2.0:1789760787.946793:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.946801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.946802:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27533312, 27537408) 0 0 00000080:00200000:2.0:1789760787.946806:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.946979:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6722, 6722] 00020000:00200000:2.0:1789760787.946987:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27533312 chunk: [27533312, 27537408) 27537408, 4096 00020000:00200000:2.0:1789760787.946991:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.947009:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6722, 6722] 00000008:00010000:2.0:1789760787.947024:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.947029:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760787.947039:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27533312, 27537408) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.947051:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27533312=0x1a42000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.947100:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.947102:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27537408 00000020:00010000:2.0:1789760787.947109:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.947111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.947113:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.947116:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.947118:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.947122:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.947128:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.947140:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760787.947684:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 27533312<>27537408 00000100:00100000:3.0:1789760787.947774:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495944192:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760787.993485:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974039 gen 2 00000008:00010000:3.0:1789760787.993533:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760787.993541:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.993552:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.993576:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760787.993605:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495944192:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760787.993806:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6722, 6722] 00000080:00200000:1.0:1789760787.993824:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760787.993830:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760787.993846:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27537408 i_blocks 55305 00000020:00200000:1.0:1789760787.993850:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760787.993860:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760787.993895:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27537408, count: 4096 00000080:00200000:1.0:1789760787.993902:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27537408, bytes: 4096 00000080:00200000:1.0:1789760787.993912:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760787.993921:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760787.993923:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27537408, 27541504) 0 0 00000080:00200000:1.0:1789760787.993928:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760787.993944:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6723, 6723] 00020000:00200000:1.0:1789760787.993952:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27537408 chunk: [27537408, 27541504) 27541504, 4096 00020000:00200000:1.0:1789760787.993956:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760787.993974:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6723, 6723] 00000008:00010000:1.0:1789760787.993991:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760787.993997:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760787.994008:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27537408, 27541504) append 0 append lockpos 0 00000080:00200000:1.0:1789760787.994020:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27537408=0x1a43000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760787.994071:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760787.994073:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27541504 00000020:00010000:1.0:1789760787.994081:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760787.994084:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.994087:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760787.994091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760787.994094:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760787.994100:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760787.994107:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760787.994122:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760787.994354:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27537408<>27541504 00000100:00100000:1.0:1789760787.994394:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760787.994412:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495944320:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.036869:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974040 gen 2 00000008:00010000:1.0:1789760788.036920:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:1.0:1789760788.036929:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.036938:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.036962:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:1.0:1789760788.036991:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495944320:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.037096:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6723, 6723] 00000080:00200000:0.0:1789760788.037114:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.037119:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.037135:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27541504 i_blocks 55305 00000020:00200000:0.0:1789760788.037138:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.037146:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.037197:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27541504, count: 4096 00000080:00200000:0.0:1789760788.037204:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27541504, bytes: 4096 00000080:00200000:0.0:1789760788.037216:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.037226:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.037228:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27541504, 27545600) 0 0 00000080:00200000:0.0:1789760788.037233:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.037251:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6724, 6724] 00020000:00200000:0.0:1789760788.037619:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27541504 chunk: [27541504, 27545600) 27545600, 4096 00020000:00200000:0.0:1789760788.037626:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.037649:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6724, 6724] 00000008:00010000:0.0:1789760788.037667:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.037672:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.037686:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27541504, 27545600) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.037703:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27541504=0x1a44000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.037764:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.037767:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27545600 00000020:00010000:0.0:1789760788.037776:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.037780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.037783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.037787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.037790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.037798:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.037805:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.037839:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760788.038755:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 27541504<>27545600 00000100:00100000:1.0:1789760788.038835:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495944448:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.099518:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974041 gen 2 00000008:00010000:1.0:1789760788.099571:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760788.099578:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.099637:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.099691:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760788.099714:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495944448:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760788.099787:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6724, 6724] 00000080:00200000:3.0:1789760788.099804:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760788.099810:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760788.099994:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27545600 i_blocks 55305 00000020:00200000:3.0:1789760788.100000:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760788.100009:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760788.100043:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27545600, count: 4096 00000080:00200000:3.0:1789760788.100051:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27545600, bytes: 4096 00000080:00200000:3.0:1789760788.100062:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760788.100071:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760788.100074:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27545600, 27549696) 0 0 00000080:00200000:3.0:1789760788.100078:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760788.100097:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6725, 6725] 00020000:00200000:3.0:1789760788.100106:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27545600 chunk: [27545600, 27549696) 27549696, 4096 00020000:00200000:3.0:1789760788.100110:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760788.100131:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6725, 6725] 00000008:00010000:3.0:1789760788.100150:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760788.100156:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760788.100190:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27545600, 27549696) append 0 append lockpos 0 00000080:00200000:3.0:1789760788.100206:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27545600=0x1a45000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760788.100333:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760788.100336:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27549696 00000020:00010000:3.0:1789760788.100345:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760788.100349:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.100352:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.100355:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760788.100358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760788.100365:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760788.100372:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760788.100456:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760788.100987:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 27545600<>27549696 00000100:00100000:2.0:1789760788.101077:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495944576:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760788.137990:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974042 gen 2 00000008:00010000:2.0:1789760788.138202:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:2.0:1789760788.138210:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760788.138222:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760788.138475:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:2.0:1789760788.138505:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495944576:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.138622:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6725, 6725] 00000080:00200000:0.0:1789760788.138638:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.138645:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.138661:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27549696 i_blocks 55305 00000020:00200000:0.0:1789760788.138665:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.138674:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.138709:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27549696, count: 4096 00000080:00200000:0.0:1789760788.138989:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27549696, bytes: 4096 00000080:00200000:0.0:1789760788.139004:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.139014:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.139017:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27549696, 27553792) 0 0 00000080:00200000:0.0:1789760788.139203:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.139221:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6726, 6726] 00020000:00200000:0.0:1789760788.139230:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27549696 chunk: [27549696, 27553792) 27553792, 4096 00020000:00200000:0.0:1789760788.139234:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.139475:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6726, 6726] 00000008:00010000:0.0:1789760788.139493:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.139502:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.139513:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27549696, 27553792) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.139529:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27549696=0x1a46000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.139659:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.139663:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27553792 00000020:00010000:0.0:1789760788.139672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.139675:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.139679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.139682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.139685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.139694:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.139701:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.139722:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760788.139966:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 27549696<>27553792 00000100:00100000:1.0:1789760788.140060:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495944704:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.196522:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974043 gen 2 00000008:00010000:1.0:1789760788.196570:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760788.196578:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.196598:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.196620:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760788.196646:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495944704:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760788.197806:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6726, 6726] 00000080:00200000:2.0:1789760788.197826:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.197833:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.197849:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27553792 i_blocks 55305 00000020:00200000:2.0:1789760788.197854:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.197864:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.197897:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27553792, count: 4096 00000080:00200000:2.0:1789760788.197905:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27553792, bytes: 4096 00000080:00200000:2.0:1789760788.197916:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.197924:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.197926:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27553792, 27557888) 0 0 00000080:00200000:2.0:1789760788.197930:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.197947:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6727, 6727] 00020000:00200000:2.0:1789760788.197955:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27553792 chunk: [27553792, 27557888) 27557888, 4096 00020000:00200000:2.0:1789760788.197959:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.197976:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6727, 6727] 00000008:00010000:2.0:1789760788.197993:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.197998:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760788.198009:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27553792, 27557888) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.198022:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27553792=0x1a47000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.198074:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.198076:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27557888 00000020:00010000:2.0:1789760788.198083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.198087:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.198090:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.198093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.198096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.198104:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.198110:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.198124:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760788.198296:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 27553792<>27557888 00000100:00100000:2.0:1789760788.198345:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760788.198364:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495944832:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760788.239892:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974044 gen 2 00000008:00010000:2.0:1789760788.239937:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:2.0:1789760788.239946:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760788.239957:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760788.239979:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:2.0:1789760788.240005:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495944832:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760788.240100:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6727, 6727] 00000080:00200000:3.0:1789760788.240117:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760788.240122:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760788.240138:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27557888 i_blocks 55305 00000020:00200000:3.0:1789760788.240142:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760788.240150:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760788.240218:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27557888, count: 4096 00000080:00200000:3.0:1789760788.240225:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27557888, bytes: 4096 00000080:00200000:3.0:1789760788.240244:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760788.240254:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760788.240267:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27557888, 27561984) 0 0 00000080:00200000:3.0:1789760788.240275:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760788.240293:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6728, 6728] 00020000:00200000:3.0:1789760788.240302:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27557888 chunk: [27557888, 27561984) 27561984, 4096 00020000:00200000:3.0:1789760788.240307:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760788.240327:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6728, 6728] 00000008:00010000:3.0:1789760788.240344:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760788.240348:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760788.240358:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27557888, 27561984) append 0 append lockpos 0 00000080:00200000:3.0:1789760788.240371:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27557888=0x1a48000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760788.240421:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760788.240424:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27561984 00000020:00010000:3.0:1789760788.240431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760788.240435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.240438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.240442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760788.240445:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760788.240451:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760788.240456:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760788.240470:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760788.240593:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 27557888<>27561984 00000100:00100000:3.0:1789760788.240627:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.240641:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495944960:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760788.286567:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974045 gen 2 00000008:00010000:3.0:1789760788.286621:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:3.0:1789760788.286630:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.286642:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.286671:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:3.0:1789760788.286701:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495944960:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760788.286830:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6728, 6728] 00000080:00200000:1.0:1789760788.286851:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.286858:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.286873:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27561984 i_blocks 55305 00000020:00200000:1.0:1789760788.286877:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.286887:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.286922:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27561984, count: 4096 00000080:00200000:1.0:1789760788.286932:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27561984, bytes: 4096 00000080:00200000:1.0:1789760788.286944:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.286954:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.286956:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27561984, 27566080) 0 0 00000080:00200000:1.0:1789760788.286961:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.286977:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6729, 6729] 00020000:00200000:1.0:1789760788.286986:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27561984 chunk: [27561984, 27566080) 27566080, 4096 00020000:00200000:1.0:1789760788.286991:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.287010:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6729, 6729] 00000008:00010000:1.0:1789760788.287026:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.287032:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760788.287053:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27561984, 27566080) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.287068:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27561984=0x1a49000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.287119:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.287122:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27566080 00000020:00010000:1.0:1789760788.287131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.287134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.287136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.287140:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.287142:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.287149:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.287155:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760788.287492:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:3.0:1789760788.287610:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 27561984<>27566080 00000100:00100000:2.0:1789760788.287685:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760788.289132:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495945088:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760788.336640:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:2.0:1789760788.336657:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760788.336669:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760788.336756:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:2.0:1789760788.336786:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495945088:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760788.337493:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6729, 6729] 00000080:00200000:1.0:1789760788.337512:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.337519:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.337536:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27566080 i_blocks 55305 00000020:00200000:1.0:1789760788.337541:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.337550:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.337643:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27566080, count: 4096 00000080:00200000:1.0:1789760788.337651:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27566080, bytes: 4096 00000080:00200000:1.0:1789760788.337662:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.337672:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.337674:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27566080, 27570176) 0 0 00000080:00200000:1.0:1789760788.337679:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.337698:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6730, 6730] 00020000:00200000:1.0:1789760788.337707:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27566080 chunk: [27566080, 27570176) 27570176, 4096 00020000:00200000:1.0:1789760788.337778:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.337800:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6730, 6730] 00000008:00010000:1.0:1789760788.337818:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.337824:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760788.338051:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27566080, 27570176) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.338068:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27566080=0x1a4a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.338139:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.338142:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27570176 00000020:00010000:1.0:1789760788.338790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.338795:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.338799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.338802:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.338806:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.338818:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.338827:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.338969:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760788.339118:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27566080<>27570176 00000100:00100000:0.0:1789760788.339204:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495945216:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.390867:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974047 gen 2 00000008:00010000:1.0:1789760788.390937:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760788.390945:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.390956:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.390978:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760788.391005:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495945216:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760788.391429:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6730, 6730] 00000080:00200000:3.0:1789760788.391447:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760788.391454:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760788.391470:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27570176 i_blocks 55305 00000020:00200000:3.0:1789760788.391474:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760788.391485:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760788.391520:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27570176, count: 4096 00000080:00200000:3.0:1789760788.391527:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27570176, bytes: 4096 00000080:00200000:3.0:1789760788.391538:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760788.391547:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760788.391549:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27570176, 27574272) 0 0 00000080:00200000:3.0:1789760788.391554:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760788.391571:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6731, 6731] 00020000:00200000:3.0:1789760788.391579:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27570176 chunk: [27570176, 27574272) 27574272, 4096 00020000:00200000:3.0:1789760788.391585:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760788.391606:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6731, 6731] 00000008:00010000:3.0:1789760788.391624:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760788.391630:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760788.391642:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27570176, 27574272) append 0 append lockpos 0 00000080:00200000:3.0:1789760788.391656:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27570176=0x1a4b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760788.391714:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760788.391717:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27574272 00000020:00010000:3.0:1789760788.391726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760788.391730:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.391733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.391736:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760788.391739:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760788.391747:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760788.391755:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760788.391771:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760788.392449:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 27570176<>27574272 00000100:00100000:2.0:1789760788.392519:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495945344:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760788.449741:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974048 gen 2 00000008:00010000:2.0:1789760788.449795:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760788.449803:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760788.449815:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760788.449844:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:2.0:1789760788.449874:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495945344:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760788.450140:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6731, 6731] 00000080:00200000:1.0:1789760788.450157:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.450163:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.450211:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27574272 i_blocks 55305 00000020:00200000:1.0:1789760788.450215:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.450225:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.450271:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27574272, count: 4096 00000080:00200000:1.0:1789760788.450278:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27574272, bytes: 4096 00000080:00200000:1.0:1789760788.450289:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.450299:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.450302:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27574272, 27578368) 0 0 00000080:00200000:1.0:1789760788.450306:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.450323:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6732, 6732] 00020000:00200000:1.0:1789760788.450331:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27574272 chunk: [27574272, 27578368) 27578368, 4096 00020000:00200000:1.0:1789760788.450336:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.450356:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6732, 6732] 00000008:00010000:1.0:1789760788.450374:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.450380:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760788.450391:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27574272, 27578368) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.450405:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27574272=0x1a4c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.450459:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.450462:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27578368 00000020:00010000:1.0:1789760788.450470:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.450474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.450477:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.450480:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.450484:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.450492:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.450500:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.450518:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760788.450641:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27574272<>27578368 00000100:00100000:1.0:1789760788.450694:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760788.450712:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495945472:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.492244:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974049 gen 2 00000008:00010000:1.0:1789760788.492303:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760788.492455:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.492468:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.492493:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760788.492525:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495945472:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760788.492691:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6732, 6732] 00000080:00200000:2.0:1789760788.492709:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.492715:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.492731:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27578368 i_blocks 55305 00000020:00200000:2.0:1789760788.492734:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.492744:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.492779:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27578368, count: 4096 00000080:00200000:2.0:1789760788.492786:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27578368, bytes: 4096 00000080:00200000:2.0:1789760788.492798:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.492808:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.492810:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27578368, 27582464) 0 0 00000080:00200000:2.0:1789760788.492814:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.492831:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6733, 6733] 00020000:00200000:2.0:1789760788.492840:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27578368 chunk: [27578368, 27582464) 27582464, 4096 00020000:00200000:2.0:1789760788.492844:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.492867:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6733, 6733] 00000008:00010000:2.0:1789760788.492890:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.492897:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760788.492909:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27578368, 27582464) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.492924:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27578368=0x1a4d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.492982:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.492985:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27582464 00000020:00010000:2.0:1789760788.492993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.492997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.493000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.493004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.493007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.493014:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.493021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.493037:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813300 00000008:00100000:2.0:1789760788.493219:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 27578368<>27582464 00000100:00100000:3.0:1789760788.494398:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495945600:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760788.538872:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974050 gen 2 00000008:00010000:3.0:1789760788.538924:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813300 00000020:00200000:3.0:1789760788.538933:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.538945:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.538968:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813300 00000100:00100000:3.0:1789760788.538998:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495945600:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.539207:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6733, 6733] 00000080:00200000:0.0:1789760788.539224:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.539230:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.539246:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27582464 i_blocks 55305 00000020:00200000:0.0:1789760788.539250:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.539270:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.539307:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27582464, count: 4096 00000080:00200000:0.0:1789760788.539315:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27582464, bytes: 4096 00000080:00200000:0.0:1789760788.539325:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.539334:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.539337:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27582464, 27586560) 0 0 00000080:00200000:0.0:1789760788.539341:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.539359:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6734, 6734] 00020000:00200000:0.0:1789760788.539367:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27582464 chunk: [27582464, 27586560) 27586560, 4096 00020000:00200000:0.0:1789760788.539372:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.539392:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6734, 6734] 00000008:00010000:0.0:1789760788.539411:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.539417:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.539429:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27582464, 27586560) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.539443:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27582464=0x1a4e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.539516:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.539520:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27586560 00000020:00010000:0.0:1789760788.539529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.539532:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.539535:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.539539:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.539542:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.539550:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.539557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.539574:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760788.539929:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 27582464<>27586560 00000100:00100000:0.0:1789760788.539980:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760788.539998:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495945728:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760788.577904:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760788.577919:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.577931:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.577952:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760788.577980:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495945728:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.578105:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6734, 6734] 00000080:00200000:0.0:1789760788.578122:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.578130:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.578145:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27586560 i_blocks 55305 00000020:00200000:0.0:1789760788.578150:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.578267:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.578304:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27586560, count: 4096 00000080:00200000:0.0:1789760788.578311:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27586560, bytes: 4096 00000080:00200000:0.0:1789760788.578321:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.578330:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.578333:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27586560, 27590656) 0 0 00000080:00200000:0.0:1789760788.578337:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.578355:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6735, 6735] 00020000:00200000:0.0:1789760788.578363:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27586560 chunk: [27586560, 27590656) 27590656, 4096 00020000:00200000:0.0:1789760788.578367:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.578389:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6735, 6735] 00000008:00010000:0.0:1789760788.578407:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.578413:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.578425:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27586560, 27590656) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.578438:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27586560=0x1a4f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.578497:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.578500:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27590656 00000020:00010000:0.0:1789760788.578508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.578512:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.578515:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.578518:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.578521:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.578528:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.578534:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.578551:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760788.578687:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 27586560<>27590656 00000100:00100000:1.0:1789760788.578960:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495945856:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.615632:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974052 gen 2 00000008:00010000:1.0:1789760788.615695:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760788.615704:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.615719:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.615742:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760788.615773:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495945856:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.615941:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6735, 6735] 00000080:00200000:0.0:1789760788.615958:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.615965:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.615982:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27590656 i_blocks 55305 00000020:00200000:0.0:1789760788.615986:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.615996:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.616031:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27590656, count: 4096 00000080:00200000:0.0:1789760788.616038:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27590656, bytes: 4096 00000080:00200000:0.0:1789760788.616050:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.616059:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.616061:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27590656, 27594752) 0 0 00000080:00200000:0.0:1789760788.616066:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.616082:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6736, 6736] 00020000:00200000:0.0:1789760788.616090:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27590656 chunk: [27590656, 27594752) 27594752, 4096 00020000:00200000:0.0:1789760788.616095:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.616115:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6736, 6736] 00000008:00010000:0.0:1789760788.616133:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.616139:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.616150:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27590656, 27594752) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.616165:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27590656=0x1a50000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.616275:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.616278:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27594752 00000020:00010000:0.0:1789760788.616287:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.616290:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.616293:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.616296:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.616299:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.616307:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.616321:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.616339:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760788.616463:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 27590656<>27594752 00000100:00100000:1.0:1789760788.616723:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760788.616743:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495945984:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760788.653466:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760788.653561:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.653573:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.653652:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760788.653683:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495945984:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.653782:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6736, 6736] 00000080:00200000:0.0:1789760788.653799:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.653807:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.653822:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27594752 i_blocks 55305 00000020:00200000:0.0:1789760788.653826:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.653836:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.653881:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27594752, count: 4096 00000080:00200000:0.0:1789760788.653889:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27594752, bytes: 4096 00000080:00200000:0.0:1789760788.653899:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.653908:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.653911:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27594752, 27598848) 0 0 00000080:00200000:0.0:1789760788.653915:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.653935:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6737, 6737] 00020000:00200000:0.0:1789760788.653943:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27594752 chunk: [27594752, 27598848) 27598848, 4096 00020000:00200000:0.0:1789760788.653948:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.653968:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6737, 6737] 00000008:00010000:0.0:1789760788.653986:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.653992:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.654003:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27594752, 27598848) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.654018:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27594752=0x1a51000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.654075:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.654078:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27598848 00000020:00010000:0.0:1789760788.654086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.654089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.654092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.654096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.654099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.654107:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.654114:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.654132:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760788.654366:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 27594752<>27598848 00000100:00100000:1.0:1789760788.654515:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495946112:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.705121:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974054 gen 2 00000008:00010000:1.0:1789760788.705242:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760788.705251:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.705857:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.705941:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760788.705974:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495946112:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.706555:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6737, 6737] 00000080:00200000:0.0:1789760788.706690:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.706698:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.706717:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27598848 i_blocks 55305 00000020:00200000:0.0:1789760788.706722:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.706733:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.706769:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27598848, count: 4096 00000080:00200000:0.0:1789760788.706777:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27598848, bytes: 4096 00000080:00200000:0.0:1789760788.706790:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.706798:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.706801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27598848, 27602944) 0 0 00000080:00200000:0.0:1789760788.706805:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.706821:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6738, 6738] 00020000:00200000:0.0:1789760788.706829:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27598848 chunk: [27598848, 27602944) 27602944, 4096 00020000:00200000:0.0:1789760788.706834:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.706916:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6738, 6738] 00000008:00010000:0.0:1789760788.706938:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.706944:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.706956:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27598848, 27602944) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.706971:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27598848=0x1a52000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.707033:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.707035:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27602944 00000020:00010000:0.0:1789760788.707045:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.707049:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.707052:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.707055:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.707059:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.707066:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.707073:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.707095:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760788.707765:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 27598848<>27602944 00000100:00100000:1.0:1789760788.707843:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495946240:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760788.774446:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760788.774463:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.774476:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.774500:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760788.774532:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495946240:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.774951:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6738, 6738] 00000080:00200000:0.0:1789760788.774965:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.774970:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.774984:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27602944 i_blocks 55305 00000020:00200000:0.0:1789760788.774987:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.774995:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.775028:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27602944, count: 4096 00000080:00200000:0.0:1789760788.775035:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27602944, bytes: 4096 00000080:00200000:0.0:1789760788.775046:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.775055:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.775058:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27602944, 27607040) 0 0 00000080:00200000:0.0:1789760788.775062:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.775079:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6739, 6739] 00020000:00200000:0.0:1789760788.775087:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27602944 chunk: [27602944, 27607040) 27607040, 4096 00020000:00200000:0.0:1789760788.775091:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.775115:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6739, 6739] 00000008:00010000:0.0:1789760788.775132:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.775138:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.775149:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27602944, 27607040) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.775163:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27602944=0x1a53000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.776051:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.776055:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27607040 00000020:00010000:0.0:1789760788.776065:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.776069:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.776072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.776076:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.776079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.776090:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.776097:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.776123:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760788.776299:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 27602944<>27607040 00000100:00100000:1.0:1789760788.776573:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495946368:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.820938:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974056 gen 2 00000008:00010000:1.0:1789760788.821004:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760788.821013:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.821024:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.821046:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760788.821077:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495946368:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.821202:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6739, 6739] 00000080:00200000:0.0:1789760788.821218:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.821225:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.821250:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27607040 i_blocks 55305 00000020:00200000:0.0:1789760788.821265:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.821274:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.821309:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27607040, count: 4096 00000080:00200000:0.0:1789760788.821316:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27607040, bytes: 4096 00000080:00200000:0.0:1789760788.821326:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.821335:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.821338:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27607040, 27611136) 0 0 00000080:00200000:0.0:1789760788.821342:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.821360:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6740, 6740] 00020000:00200000:0.0:1789760788.821368:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27607040 chunk: [27607040, 27611136) 27611136, 4096 00020000:00200000:0.0:1789760788.821372:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.821392:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6740, 6740] 00000008:00010000:0.0:1789760788.821410:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.821416:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.821428:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27607040, 27611136) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.821444:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27607040=0x1a54000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.821501:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.821503:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27611136 00000020:00010000:0.0:1789760788.821510:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.821514:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.821517:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.821520:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.821523:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.821530:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.821537:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.821553:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760788.821691:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 27607040<>27611136 00000100:00100000:1.0:1789760788.822204:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760788.822226:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495946496:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760788.855845:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974057 gen 2 00000008:00010000:1.0:1789760788.855897:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760788.855905:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.855916:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.855938:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760788.855966:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495946496:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.856326:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6740, 6740] 00000080:00200000:0.0:1789760788.856345:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.856351:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.856367:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27611136 i_blocks 55305 00000020:00200000:0.0:1789760788.856371:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.856381:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.856416:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27611136, count: 4096 00000080:00200000:0.0:1789760788.856424:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27611136, bytes: 4096 00000080:00200000:0.0:1789760788.856435:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.856444:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.856447:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27611136, 27615232) 0 0 00000080:00200000:0.0:1789760788.856452:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.856469:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6741, 6741] 00020000:00200000:0.0:1789760788.856478:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27611136 chunk: [27611136, 27615232) 27615232, 4096 00020000:00200000:0.0:1789760788.856483:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.856504:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6741, 6741] 00000008:00010000:0.0:1789760788.856521:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.856527:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760788.856538:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27611136, 27615232) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.856553:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27611136=0x1a55000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.856678:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.856682:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27615232 00000020:00010000:0.0:1789760788.856690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.856694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.856698:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.856702:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.856705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.856712:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.856719:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.856735:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760788.856933:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 27611136<>27615232 00000100:00100000:1.0:1789760788.857398:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495946624:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760788.893601:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760788.893617:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.893629:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.893695:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760788.893726:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495946624:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760788.894154:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6741, 6741] 00000080:00200000:0.0:1789760788.894210:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.894217:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.894366:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27615232 i_blocks 55305 00000020:00200000:1.0:1789760788.894372:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.894382:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.894416:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27615232, count: 4096 00000080:00200000:1.0:1789760788.894423:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27615232, bytes: 4096 00000080:00200000:1.0:1789760788.894434:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.894443:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.894446:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27615232, 27619328) 0 0 00000080:00200000:1.0:1789760788.894451:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.894468:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6742, 6742] 00020000:00200000:1.0:1789760788.894476:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27615232 chunk: [27615232, 27619328) 27619328, 4096 00020000:00200000:1.0:1789760788.894481:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.894500:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6742, 6742] 00000008:00010000:1.0:1789760788.894518:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.894523:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760788.894534:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27615232, 27619328) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.894548:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27615232=0x1a56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.894668:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.894671:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27619328 00000020:00010000:1.0:1789760788.894680:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.894684:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.894687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.894691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.894694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.894701:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.894708:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.894723:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760788.894976:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27615232<>27619328 00000100:00100000:0.0:1789760788.895049:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495946752:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760788.939568:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760788.947002:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974059 gen 2 00000008:00010000:1.0:1789760788.947067:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760788.947075:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.947086:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.947116:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760788.947145:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495946752:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760788.947425:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6742, 6742] 00000080:00200000:2.0:1789760788.947443:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.947450:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.947466:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27619328 i_blocks 55305 00000020:00200000:2.0:1789760788.947470:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.947480:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.947513:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27619328, count: 4096 00000080:00200000:2.0:1789760788.947522:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27619328, bytes: 4096 00000080:00200000:2.0:1789760788.947533:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.947541:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.947543:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27619328, 27623424) 0 0 00000080:00200000:2.0:1789760788.947548:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.947564:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6743, 6743] 00020000:00200000:2.0:1789760788.947572:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27619328 chunk: [27619328, 27623424) 27623424, 4096 00020000:00200000:2.0:1789760788.947646:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.947668:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6743, 6743] 00000008:00010000:2.0:1789760788.947687:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.947693:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760788.947705:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27619328, 27623424) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.947720:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27619328=0x1a57000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.947790:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.947793:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27623424 00000020:00010000:2.0:1789760788.947801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.947804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.947807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.947811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.947814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.947822:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.947829:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.947845:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760788.947976:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267480 - object 0x240000400:4 offset 27619328<>27623424 00000100:00100000:3.0:1789760788.948203:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495946880:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760788.993572:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760788.993587:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.993599:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.993623:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760788.993654:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495946880:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760788.993865:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6743, 6743] 00000080:00200000:2.0:1789760788.993883:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.993889:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.993904:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27623424 i_blocks 55305 00000020:00200000:2.0:1789760788.993908:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.993918:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.993954:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27623424, count: 4096 00000080:00200000:2.0:1789760788.993961:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27623424, bytes: 4096 00000080:00200000:2.0:1789760788.993971:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.993980:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.993982:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27623424, 27627520) 0 0 00000080:00200000:2.0:1789760788.993988:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.994004:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6744, 6744] 00020000:00200000:2.0:1789760788.994013:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27623424 chunk: [27623424, 27627520) 27627520, 4096 00020000:00200000:2.0:1789760788.994018:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.994037:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6744, 6744] 00000008:00010000:2.0:1789760788.994056:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.994062:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760788.994074:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27623424, 27627520) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.994088:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27623424=0x1a58000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.994147:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.994150:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27627520 00000020:00010000:2.0:1789760788.994160:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.994163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.994166:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.996363:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.996368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.996386:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.996398:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.996436:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760788.996568:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264a80 - object 0x240000400:4 offset 27623424<>27627520 00000100:00100000:3.0:1789760788.997277:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760788.997376:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947008:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760789.047126:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974061 gen 2 00000008:00010000:3.0:1789760789.047222:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760789.047240:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.047264:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.047292:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760789.047322:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947008:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760789.047435:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6744, 6744] 00000080:00200000:1.0:1789760789.047453:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.047458:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.047474:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27627520 i_blocks 55305 00000020:00200000:1.0:1789760789.047477:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.047488:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.047527:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27627520, count: 4096 00000080:00200000:1.0:1789760789.047535:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27627520, bytes: 4096 00000080:00200000:1.0:1789760789.047546:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.047555:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.047557:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27627520, 27631616) 0 0 00000080:00200000:1.0:1789760789.047563:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.047591:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6745, 6745] 00020000:00200000:1.0:1789760789.047600:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27627520 chunk: [27627520, 27631616) 27631616, 4096 00020000:00200000:1.0:1789760789.047604:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.047624:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6745, 6745] 00000008:00010000:1.0:1789760789.047643:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.047649:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760789.047661:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27627520, 27631616) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.047678:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27627520=0x1a59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.047734:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.047737:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27631616 00000020:00010000:1.0:1789760789.047745:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.047749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.047752:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.047755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.047758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.047765:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.047772:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.047787:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760789.048101:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27627520<>27631616 00000100:00100000:1.0:1789760789.048154:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760789.048218:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495947136:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.085030:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974062 gen 2 00000008:00010000:1.0:1789760789.085089:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760789.085099:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.085110:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.085134:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760789.085165:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495947136:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.086367:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6745, 6745] 00000080:00200000:0.0:1789760789.086390:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.086398:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.086417:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27631616 i_blocks 55305 00000020:00200000:0.0:1789760789.086422:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.086435:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.086473:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27631616, count: 4096 00000080:00200000:0.0:1789760789.086481:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27631616, bytes: 4096 00000080:00200000:0.0:1789760789.086495:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.086505:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.086508:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27631616, 27635712) 0 0 00000080:00200000:0.0:1789760789.086512:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.086530:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6746, 6746] 00020000:00200000:0.0:1789760789.086537:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27631616 chunk: [27631616, 27635712) 27635712, 4096 00020000:00200000:0.0:1789760789.086542:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.086564:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6746, 6746] 00000008:00010000:0.0:1789760789.086584:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.086590:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.086603:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27631616, 27635712) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.086618:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27631616=0x1a5a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.086685:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.086688:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27635712 00000020:00010000:0.0:1789760789.086702:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.086706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.086709:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.086713:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.086716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.086724:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.086730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.086752:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760789.086890:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27631616<>27635712 00000100:00100000:1.0:1789760789.086974:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495947264:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.122403:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974063 gen 2 00000008:00010000:1.0:1789760789.122454:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760789.122462:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.122473:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.122493:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760789.122524:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495947264:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.122931:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6746, 6746] 00000080:00200000:0.0:1789760789.122947:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.122954:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.122970:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27635712 i_blocks 55305 00000020:00200000:0.0:1789760789.122974:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.122984:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.123018:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27635712, count: 4096 00000080:00200000:0.0:1789760789.123026:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27635712, bytes: 4096 00000080:00200000:0.0:1789760789.123094:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.123105:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.123107:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27635712, 27639808) 0 0 00000080:00200000:0.0:1789760789.123112:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.123132:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6747, 6747] 00020000:00200000:0.0:1789760789.123140:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27635712 chunk: [27635712, 27639808) 27639808, 4096 00020000:00200000:0.0:1789760789.123144:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.123164:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6747, 6747] 00000008:00010000:0.0:1789760789.123206:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.123213:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.123224:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27635712, 27639808) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.123839:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27635712=0x1a5b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.123918:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.123921:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27639808 00000020:00010000:0.0:1789760789.123931:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.123935:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.123938:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.123941:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.123944:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.123953:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.123961:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.123987:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760789.124551:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 27635712<>27639808 00000100:00100000:0.0:1789760789.124679:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495947392:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.187430:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974064 gen 2 00000008:00010000:1.0:1789760789.187485:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760789.187494:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.187506:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.187587:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760789.187616:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495947392:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760789.190105:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6747, 6747] 00000080:00200000:2.0:1789760789.190132:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.190139:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.190160:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27639808 i_blocks 55305 00000020:00200000:2.0:1789760789.190166:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.190197:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.190232:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27639808, count: 4096 00000080:00200000:2.0:1789760789.190241:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27639808, bytes: 4096 00000080:00200000:2.0:1789760789.190265:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.190276:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.190279:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27639808, 27643904) 0 0 00000080:00200000:2.0:1789760789.190283:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.190300:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6748, 6748] 00020000:00200000:2.0:1789760789.190309:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27639808 chunk: [27639808, 27643904) 27643904, 4096 00020000:00200000:2.0:1789760789.190314:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.190336:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6748, 6748] 00000008:00010000:2.0:1789760789.190641:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.190649:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760789.190664:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27639808, 27643904) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.190679:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27639808=0x1a5c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.190750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.190754:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27643904 00000020:00010000:2.0:1789760789.190762:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.190765:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.190769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.190772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.190901:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.190910:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.190918:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.190949:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760789.191085:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 27639808<>27643904 00000100:00100000:3.0:1789760789.192803:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947520:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760789.230547:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974065 gen 2 00000008:00010000:2.0:1789760789.230650:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760789.230659:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760789.230669:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760789.230740:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760789.230770:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947520:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760789.230870:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6748, 6748] 00000080:00200000:3.0:1789760789.230888:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.230895:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.230989:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27643904 i_blocks 55305 00000020:00200000:2.0:1789760789.230994:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.231003:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.231034:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27643904, count: 4096 00000080:00200000:2.0:1789760789.231041:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27643904, bytes: 4096 00000080:00200000:2.0:1789760789.231050:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.231059:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.231061:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27643904, 27648000) 0 0 00000080:00200000:2.0:1789760789.231065:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.231081:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6749, 6749] 00020000:00200000:2.0:1789760789.231152:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27643904 chunk: [27643904, 27648000) 27648000, 4096 00020000:00200000:2.0:1789760789.231158:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.231209:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6749, 6749] 00000008:00010000:2.0:1789760789.231229:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.231236:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760789.231262:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27643904, 27648000) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.231278:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27643904=0x1a5d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.231335:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.231338:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27648000 00000020:00010000:2.0:1789760789.231348:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.231351:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.231354:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.231358:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.231361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.231368:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.231374:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.231425:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760789.231613:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 27643904<>27648000 00000100:00100000:2.0:1789760789.231657:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760789.231675:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947648:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760789.293746:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974066 gen 2 00000008:00010000:2.0:1789760789.293793:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760789.293802:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760789.293812:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760789.293889:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760789.293918:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947648:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760789.294352:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6749, 6749] 00000080:00200000:3.0:1789760789.294371:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.294379:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.294396:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27648000 i_blocks 55305 00000020:00200000:3.0:1789760789.294400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.294411:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.294775:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27648000, count: 4096 00000080:00200000:3.0:1789760789.294785:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27648000, bytes: 4096 00000080:00200000:3.0:1789760789.294799:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.294808:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.294810:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27648000, 27652096) 0 0 00000080:00200000:3.0:1789760789.294814:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.294833:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6750, 6750] 00020000:00200000:3.0:1789760789.294842:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27648000 chunk: [27648000, 27652096) 27652096, 4096 00020000:00200000:3.0:1789760789.294846:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.294927:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6750, 6750] 00000008:00010000:3.0:1789760789.294948:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.294953:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760789.294967:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27648000, 27652096) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.294982:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27648000=0x1a5e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.295053:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.295056:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27652096 00000020:00010000:3.0:1789760789.295064:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.295068:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.295071:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.295074:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.295077:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.295085:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.295093:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.295118:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760789.295359:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147b80 - object 0x240000400:4 offset 27648000<>27652096 00000100:00100000:2.0:1789760789.295452:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947776:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760789.332399:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974067 gen 2 00000008:00010000:2.0:1789760789.332455:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760789.332465:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760789.332477:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760789.332501:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760789.332534:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947776:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760789.332667:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6750, 6750] 00000080:00200000:3.0:1789760789.332684:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.332690:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.332707:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27652096 i_blocks 55305 00000020:00200000:3.0:1789760789.332710:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.332719:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.332756:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27652096, count: 4096 00000080:00200000:3.0:1789760789.332764:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27652096, bytes: 4096 00000080:00200000:3.0:1789760789.332787:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.332797:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.332800:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27652096, 27656192) 0 0 00000080:00200000:3.0:1789760789.332805:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.332823:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6751, 6751] 00020000:00200000:3.0:1789760789.332831:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27652096 chunk: [27652096, 27656192) 27656192, 4096 00020000:00200000:3.0:1789760789.332836:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.332856:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6751, 6751] 00000008:00010000:3.0:1789760789.332873:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.332879:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760789.332890:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27652096, 27656192) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.332903:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27652096=0x1a5f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.332961:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.332964:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27656192 00000020:00010000:3.0:1789760789.332973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.332977:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.332980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.332984:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.332987:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.332994:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.333003:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.333019:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760789.333143:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146a00 - object 0x240000400:4 offset 27652096<>27656192 00000100:00100000:2.0:1789760789.333406:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760789.333425:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947904:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760789.370768:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974068 gen 2 00000008:00010000:2.0:1789760789.370896:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760789.370906:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760789.370921:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760789.370949:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760789.370978:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495947904:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760789.371106:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6751, 6751] 00000080:00200000:3.0:1789760789.371124:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.371130:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.371146:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27656192 i_blocks 55305 00000020:00200000:3.0:1789760789.371149:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.371158:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.371231:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27656192, count: 4096 00000080:00200000:3.0:1789760789.371238:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27656192, bytes: 4096 00000080:00200000:3.0:1789760789.371260:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.371269:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.371271:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27656192, 27660288) 0 0 00000080:00200000:3.0:1789760789.371275:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.371291:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6752, 6752] 00020000:00200000:3.0:1789760789.371300:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27656192 chunk: [27656192, 27660288) 27660288, 4096 00020000:00200000:3.0:1789760789.371304:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.371325:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6752, 6752] 00000008:00010000:3.0:1789760789.371343:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.371349:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760789.371361:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27656192, 27660288) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.371605:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27656192=0x1a60000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.371672:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.371675:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27660288 00000020:00010000:3.0:1789760789.371685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.371688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.371691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.371694:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.371697:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.371705:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.371713:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.371733:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760789.376082:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 27656192<>27660288 00000100:00100000:3.0:1789760789.376213:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760789.376241:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495948032:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760789.389335:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495948288:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760789.389500:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1789760789.389509:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2358:1876699495948160:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760789.389552:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495948416:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760789.396299:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2358:1876699495948160:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760789.396331:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495948288:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760789.407805:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495948416:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760789.428072:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974069 gen 2 00000008:00010000:3.0:1789760789.428122:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:3.0:1789760789.428131:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.428143:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.428218:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000080:00200000:2.0:1789760789.428388:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6752, 6752] 00000080:00200000:2.0:1789760789.428408:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.428414:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.428431:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27660288 i_blocks 55305 00000020:00200000:2.0:1789760789.428434:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.428444:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.428477:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27660288, count: 4096 00000080:00200000:2.0:1789760789.428486:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27660288, bytes: 4096 00000080:00200000:2.0:1789760789.428496:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.428506:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.428508:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27660288, 27664384) 0 0 00000080:00200000:2.0:1789760789.428512:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.428529:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6753, 6753] 00020000:00200000:2.0:1789760789.428538:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27660288 chunk: [27660288, 27664384) 27664384, 4096 00020000:00200000:2.0:1789760789.428542:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.428562:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6753, 6753] 00000008:00010000:2.0:1789760789.428582:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.428588:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760789.428601:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27660288, 27664384) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.428617:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27660288=0x1a61000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.428678:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.428681:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27664384 00000020:00010000:2.0:1789760789.428689:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.428693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.428696:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.428700:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.428704:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.428712:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.428719:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.428740:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760789.428868:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 27660288<>27664384 00000100:00100000:2.0:1789760789.428915:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495948544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760789.429089:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495948032:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760789.476029:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974070 gen 2 00000008:00010000:2.0:1789760789.476078:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760789.476086:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760789.476096:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760789.476117:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000080:00200000:3.0:1789760789.476149:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6753, 6753] 00000100:00100000:2.0:1789760789.476151:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495948544:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760789.476407:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.476414:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.476434:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27664384 i_blocks 55305 00000020:00200000:3.0:1789760789.476439:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.476451:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.476490:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27664384, count: 4096 00000080:00200000:3.0:1789760789.476498:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27664384, bytes: 4096 00000080:00200000:3.0:1789760789.476513:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.476521:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.476523:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27664384, 27668480) 0 0 00000080:00200000:3.0:1789760789.476528:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.476544:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6754, 6754] 00020000:00200000:3.0:1789760789.476554:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27664384 chunk: [27664384, 27668480) 27668480, 4096 00020000:00200000:3.0:1789760789.476558:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.476647:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6754, 6754] 00000008:00010000:3.0:1789760789.476666:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.476672:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760789.476684:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27664384, 27668480) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.476699:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27664384=0x1a62000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.476758:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.476761:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27668480 00000020:00010000:3.0:1789760789.476770:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.476773:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.476777:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.476780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.476783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.476791:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.476798:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.476815:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760789.476937:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146d80 - object 0x240000400:4 offset 27664384<>27668480 00000100:00100000:2.0:1789760789.477010:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495948672:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760789.533815:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974071 gen 2 00000008:00010000:2.0:1789760789.533867:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760789.533876:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760789.533886:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760789.533910:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760789.533940:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495948672:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.534051:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6754, 6754] 00000080:00200000:0.0:1789760789.534078:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.534084:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.534100:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27668480 i_blocks 55305 00000020:00200000:0.0:1789760789.534103:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.534117:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.534156:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27668480, count: 4096 00000080:00200000:0.0:1789760789.534163:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27668480, bytes: 4096 00000080:00200000:0.0:1789760789.534203:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.534213:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.534215:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27668480, 27672576) 0 0 00000080:00200000:0.0:1789760789.534220:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.534237:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6755, 6755] 00020000:00200000:0.0:1789760789.534245:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27668480 chunk: [27668480, 27672576) 27672576, 4096 00020000:00200000:0.0:1789760789.534259:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.534280:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6755, 6755] 00000008:00010000:0.0:1789760789.534297:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.534302:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.534313:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27668480, 27672576) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.534327:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27668480=0x1a63000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.534383:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.534386:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27672576 00000020:00010000:0.0:1789760789.534395:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.534398:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.534401:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.534404:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.534407:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.534416:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.534422:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.534438:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760789.534565:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 27668480<>27672576 00000100:00100000:1.0:1789760789.534676:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.534697:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495948800:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760789.576074:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760789.576090:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.576100:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.576122:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760789.576153:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495948800:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.576479:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6755, 6755] 00000080:00200000:0.0:1789760789.576497:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.576503:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.576518:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27672576 i_blocks 55305 00000020:00200000:0.0:1789760789.576521:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.576531:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.576574:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27672576, count: 4096 00000080:00200000:0.0:1789760789.576582:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27672576, bytes: 4096 00000080:00200000:0.0:1789760789.576592:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.576601:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.576603:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27672576, 27676672) 0 0 00000080:00200000:0.0:1789760789.576607:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.576624:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6756, 6756] 00020000:00200000:0.0:1789760789.576633:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27672576 chunk: [27672576, 27676672) 27676672, 4096 00020000:00200000:0.0:1789760789.576637:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.576659:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6756, 6756] 00000008:00010000:0.0:1789760789.576678:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.576684:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.576696:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27672576, 27676672) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.576710:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27672576=0x1a64000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.576770:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.576773:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27676672 00000020:00010000:0.0:1789760789.576782:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.576785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.576788:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.576792:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.576794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.576803:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.576810:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.576828:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760789.576953:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 27672576<>27676672 00000100:00100000:1.0:1789760789.577023:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495948928:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.620032:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974073 gen 2 00000008:00010000:1.0:1789760789.620100:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760789.620109:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.620118:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.620138:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000080:00200000:0.0:1789760789.620287:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6756, 6756] 00000080:00200000:0.0:1789760789.620304:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.620310:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.620326:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27676672 i_blocks 55305 00000020:00200000:0.0:1789760789.620330:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.620340:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.620375:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27676672, count: 4096 00000080:00200000:0.0:1789760789.620383:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27676672, bytes: 4096 00000080:00200000:0.0:1789760789.620394:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.620403:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.620405:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27676672, 27680768) 0 0 00000080:00200000:0.0:1789760789.620410:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.620428:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6757, 6757] 00020000:00200000:0.0:1789760789.620446:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27676672 chunk: [27676672, 27680768) 27680768, 4096 00020000:00200000:0.0:1789760789.620452:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.620476:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6757, 6757] 00000008:00010000:0.0:1789760789.620495:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.620501:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.620514:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27676672, 27680768) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.620529:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27676672=0x1a65000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.620701:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.620704:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27680768 00000020:00010000:0.0:1789760789.620713:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.620717:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.620721:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.620724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.620728:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.620736:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.620743:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760789.620797:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495948928:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760789.620912:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:1.0:1789760789.621041:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27676672<>27680768 00000100:00100000:1.0:1789760789.621083:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.621099:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949056:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.661970:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974074 gen 2 00000008:00010000:1.0:1789760789.662023:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760789.662031:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.662043:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.662100:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760789.662358:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949056:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.662450:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6757, 6757] 00000080:00200000:0.0:1789760789.662466:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.662472:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.662487:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27680768 i_blocks 55305 00000020:00200000:0.0:1789760789.662490:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.662499:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.662532:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27680768, count: 4096 00000080:00200000:0.0:1789760789.662538:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27680768, bytes: 4096 00000080:00200000:0.0:1789760789.662547:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.662555:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.662557:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27680768, 27684864) 0 0 00000080:00200000:0.0:1789760789.662561:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.662636:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6758, 6758] 00020000:00200000:0.0:1789760789.662645:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27680768 chunk: [27680768, 27684864) 27684864, 4096 00020000:00200000:0.0:1789760789.662650:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.662670:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6758, 6758] 00000008:00010000:0.0:1789760789.662687:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.662693:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.662705:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27680768, 27684864) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.662719:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27680768=0x1a66000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.662777:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.662780:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27684864 00000020:00010000:0.0:1789760789.662787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.662791:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.662794:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.662798:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.662801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.662808:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.662815:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.662832:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760789.663010:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 27680768<>27684864 00000100:00100000:1.0:1789760789.663070:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949184:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.700514:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974075 gen 2 00000008:00010000:1.0:1789760789.700580:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760789.700590:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.700601:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.700622:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760789.700652:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949184:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.700807:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6758, 6758] 00000080:00200000:0.0:1789760789.700824:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.700829:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.700845:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27684864 i_blocks 55305 00000020:00200000:0.0:1789760789.700849:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.700859:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.700893:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27684864, count: 4096 00000080:00200000:0.0:1789760789.700901:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27684864, bytes: 4096 00000080:00200000:0.0:1789760789.700910:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.700919:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.700921:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27684864, 27688960) 0 0 00000080:00200000:0.0:1789760789.700926:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.700943:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6759, 6759] 00020000:00200000:0.0:1789760789.700951:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27684864 chunk: [27684864, 27688960) 27688960, 4096 00020000:00200000:0.0:1789760789.700956:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.700977:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6759, 6759] 00000008:00010000:0.0:1789760789.700995:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.701002:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.701013:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27684864, 27688960) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.701028:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27684864=0x1a67000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.701086:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.701090:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27688960 00000020:00010000:0.0:1789760789.701100:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.701104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.701107:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.701111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.701114:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.701122:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.701129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.701145:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760789.701345:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 27684864<>27688960 00000100:00100000:1.0:1789760789.701641:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.701660:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949312:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.739917:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974076 gen 2 00000008:00010000:1.0:1789760789.739972:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760789.739980:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.739991:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.740079:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760789.740110:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949312:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.740232:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6759, 6759] 00000080:00200000:0.0:1789760789.740258:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.740264:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.740281:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27688960 i_blocks 55305 00000020:00200000:0.0:1789760789.740285:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.740294:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.740330:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27688960, count: 4096 00000080:00200000:0.0:1789760789.740338:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27688960, bytes: 4096 00000080:00200000:0.0:1789760789.740352:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.740360:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.740363:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27688960, 27693056) 0 0 00000080:00200000:0.0:1789760789.740367:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.740386:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6760, 6760] 00020000:00200000:0.0:1789760789.740395:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27688960 chunk: [27688960, 27693056) 27693056, 4096 00020000:00200000:0.0:1789760789.740399:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.740422:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6760, 6760] 00000008:00010000:0.0:1789760789.740442:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.740447:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.740739:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27688960, 27693056) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.740883:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27688960=0x1a68000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.740948:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.740951:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27693056 00000020:00010000:0.0:1789760789.740960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.740963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.740966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.740969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.740971:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.740979:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.740986:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.741002:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760789.741132:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 27688960<>27693056 00000100:00100000:1.0:1789760789.741288:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949440:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760789.781489:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760789.781506:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.781518:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.781540:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760789.781579:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949440:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.781794:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6760, 6760] 00000080:00200000:0.0:1789760789.781811:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.781818:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.781833:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27693056 i_blocks 55305 00000020:00200000:0.0:1789760789.781837:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.781845:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.781880:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27693056, count: 4096 00000080:00200000:0.0:1789760789.781888:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27693056, bytes: 4096 00000080:00200000:0.0:1789760789.781897:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.781905:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.781908:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27693056, 27697152) 0 0 00000080:00200000:0.0:1789760789.781912:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.781928:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6761, 6761] 00020000:00200000:0.0:1789760789.781937:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27693056 chunk: [27693056, 27697152) 27697152, 4096 00020000:00200000:0.0:1789760789.781941:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.781961:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6761, 6761] 00000008:00010000:0.0:1789760789.781978:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.781984:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.781997:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27693056, 27697152) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.782011:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27693056=0x1a69000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.782071:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.782074:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27697152 00000020:00010000:0.0:1789760789.782083:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.782086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.782089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.782093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.782097:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.782104:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.782111:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.782129:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760789.782300:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27693056<>27697152 00000100:00100000:1.0:1789760789.782447:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.782464:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949568:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.814641:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974077 gen 2 00000100:00100000:1.0:1789760789.814685:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd211837800 x1876699495949568/t4294974078(4294974078) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760805 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760789.814734:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760789.814743:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.814763:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.814781:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760789.814805:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949568:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.815373:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6761, 6761] 00000080:00200000:0.0:1789760789.815391:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.815397:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.815414:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27697152 i_blocks 55305 00000020:00200000:0.0:1789760789.815418:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.815428:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.815463:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27697152, count: 4096 00000080:00200000:0.0:1789760789.815470:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27697152, bytes: 4096 00000080:00200000:0.0:1789760789.815481:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.815490:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.815492:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27697152, 27701248) 0 0 00000080:00200000:0.0:1789760789.815497:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.815516:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6762, 6762] 00020000:00200000:0.0:1789760789.815524:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27697152 chunk: [27697152, 27701248) 27701248, 4096 00020000:00200000:0.0:1789760789.815529:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.815551:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6762, 6762] 00000008:00010000:0.0:1789760789.815585:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.815592:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.815606:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27697152, 27701248) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.815622:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27697152=0x1a6a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.815681:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.815684:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27701248 00000020:00010000:0.0:1789760789.815693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.815696:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.815699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.815703:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.815706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.815714:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.815721:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.815740:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760789.815987:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 27697152<>27701248 00000100:00100000:1.0:1789760789.816559:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949696:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.867895:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974079 gen 2 00000008:00010000:1.0:1789760789.867966:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760789.867975:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.867985:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.868005:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760789.868037:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949696:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.868196:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6762, 6762] 00000080:00200000:0.0:1789760789.868213:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.868220:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.868307:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27701248 i_blocks 55305 00000020:00200000:0.0:1789760789.868312:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.868321:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.868424:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27701248, count: 4096 00000080:00200000:0.0:1789760789.868433:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27701248, bytes: 4096 00000080:00200000:0.0:1789760789.868444:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.868453:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.868456:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27701248, 27705344) 0 0 00000080:00200000:0.0:1789760789.868461:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.868478:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6763, 6763] 00020000:00200000:0.0:1789760789.868487:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27701248 chunk: [27701248, 27705344) 27705344, 4096 00020000:00200000:0.0:1789760789.868491:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.868513:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6763, 6763] 00000008:00010000:0.0:1789760789.868530:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.868537:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.868549:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27701248, 27705344) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.868563:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27701248=0x1a6b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.868684:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.868686:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27705344 00000020:00010000:0.0:1789760789.868695:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.868699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.868702:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.868705:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.868708:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.868715:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.868723:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.868742:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760789.869104:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 27701248<>27705344 00000100:00100000:1.0:1789760789.869426:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.869449:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949824:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760789.915204:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974080 gen 2 00000008:00010000:1.0:1789760789.915372:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760789.915382:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.915394:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.915416:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760789.915450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949824:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.915871:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6763, 6763] 00000080:00200000:0.0:1789760789.915889:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.915896:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.915913:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27705344 i_blocks 55305 00000020:00200000:0.0:1789760789.915917:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.915926:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.915962:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27705344, count: 4096 00000080:00200000:0.0:1789760789.915969:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27705344, bytes: 4096 00000080:00200000:0.0:1789760789.915979:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.915988:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.915990:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27705344, 27709440) 0 0 00000080:00200000:0.0:1789760789.915995:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.916012:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6764, 6764] 00020000:00200000:0.0:1789760789.916020:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27705344 chunk: [27705344, 27709440) 27709440, 4096 00020000:00200000:0.0:1789760789.916025:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.916046:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6764, 6764] 00000008:00010000:0.0:1789760789.916065:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.916071:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.916083:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27705344, 27709440) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.916096:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27705344=0x1a6c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.916152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.916155:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27709440 00000020:00010000:0.0:1789760789.916163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.916222:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.916240:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.916243:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.916254:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.916261:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.916268:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.916286:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760789.916413:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 27705344<>27709440 00000100:00100000:1.0:1789760789.916787:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949952:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760789.967841:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760789.977601:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974081 gen 2 00000008:00010000:1.0:1789760789.977654:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760789.977663:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.977674:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.977693:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760789.977721:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495949952:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760789.977926:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6764, 6764] 00000080:00200000:0.0:1789760789.977942:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.977949:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.977967:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27709440 i_blocks 55305 00000020:00200000:0.0:1789760789.977971:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.977981:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.978018:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27709440, count: 4096 00000080:00200000:0.0:1789760789.978025:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27709440, bytes: 4096 00000080:00200000:0.0:1789760789.978034:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.978043:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.978046:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27709440, 27713536) 0 0 00000080:00200000:0.0:1789760789.978050:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.978068:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6765, 6765] 00020000:00200000:0.0:1789760789.978076:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27709440 chunk: [27709440, 27713536) 27713536, 4096 00020000:00200000:0.0:1789760789.978082:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.978105:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6765, 6765] 00000008:00010000:0.0:1789760789.978124:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.978130:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760789.978143:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27709440, 27713536) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.978159:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27709440=0x1a6d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.978264:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.978267:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27713536 00000020:00010000:0.0:1789760789.978276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.978279:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.978282:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.978285:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.978288:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.978295:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.978302:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.978318:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760789.978437:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 27709440<>27713536 00000100:00100000:1.0:1789760789.978627:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.978643:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495950080:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760790.018416:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760790.018434:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.018446:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.018469:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760790.018501:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495950080:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.018787:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6765, 6765] 00000080:00200000:0.0:1789760790.018804:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.018811:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.018827:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27713536 i_blocks 55305 00000020:00200000:0.0:1789760790.018831:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.018841:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.018876:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27713536, count: 4096 00000080:00200000:0.0:1789760790.018883:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27713536, bytes: 4096 00000080:00200000:0.0:1789760790.018895:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.018904:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.018906:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27713536, 27717632) 0 0 00000080:00200000:0.0:1789760790.018911:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.018929:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6766, 6766] 00020000:00200000:0.0:1789760790.018938:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27713536 chunk: [27713536, 27717632) 27717632, 4096 00020000:00200000:0.0:1789760790.018942:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.018963:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6766, 6766] 00000008:00010000:0.0:1789760790.018982:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.018988:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.019000:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27713536, 27717632) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.019017:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27713536=0x1a6e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.019076:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.019079:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27717632 00000020:00010000:0.0:1789760790.019088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.019092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.019095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.019099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.019102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.019110:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.019118:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.019135:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760790.019312:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 27713536<>27717632 00000100:00100000:1.0:1789760790.019507:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495950208:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.073631:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974083 gen 2 00000008:00010000:1.0:1789760790.073700:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760790.073710:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.073721:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.073744:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000080:00200000:0.0:1789760790.073950:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6766, 6766] 00000080:00200000:0.0:1789760790.073968:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.073974:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.073990:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27717632 i_blocks 55305 00000020:00200000:0.0:1789760790.073994:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.074004:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.074039:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27717632, count: 4096 00000080:00200000:0.0:1789760790.074048:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27717632, bytes: 4096 00000080:00200000:0.0:1789760790.074058:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.074067:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.074069:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27717632, 27721728) 0 0 00000080:00200000:0.0:1789760790.074074:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.074092:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6767, 6767] 00020000:00200000:0.0:1789760790.074100:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27717632 chunk: [27717632, 27721728) 27721728, 4096 00020000:00200000:0.0:1789760790.074105:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.074126:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6767, 6767] 00000008:00010000:0.0:1789760790.074145:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.074151:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.074162:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27717632, 27721728) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.074283:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27717632=0x1a6f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.074436:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.074440:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27721728 00000020:00010000:0.0:1789760790.074449:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.074452:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.074456:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.074459:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.074462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.074471:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.074478:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760790.075671:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495950208:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760790.075732:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:1.0:1789760790.076023:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27717632<>27721728 00000100:00100000:1.0:1789760790.076097:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495950336:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760790.106862:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760790.106877:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.106889:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.106910:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760790.106945:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495950336:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.107042:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6767, 6767] 00000080:00200000:0.0:1789760790.107059:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.107065:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.107081:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27721728 i_blocks 55305 00000020:00200000:0.0:1789760790.107085:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.107094:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.107126:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27721728, count: 4096 00000080:00200000:0.0:1789760790.107133:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27721728, bytes: 4096 00000080:00200000:0.0:1789760790.107144:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.107154:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.107157:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27721728, 27725824) 0 0 00000080:00200000:0.0:1789760790.107161:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.107198:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6768, 6768] 00020000:00200000:0.0:1789760790.107207:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27721728 chunk: [27721728, 27725824) 27725824, 4096 00020000:00200000:0.0:1789760790.107212:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.107241:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6768, 6768] 00000008:00010000:0.0:1789760790.107269:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.107275:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.107287:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27721728, 27725824) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.107300:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27721728=0x1a70000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.107634:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.107638:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27725824 00000020:00010000:0.0:1789760790.107647:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.107651:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.107655:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.107659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.107662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.107670:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.107676:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.107693:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760790.107809:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 27721728<>27725824 00000100:00100000:1.0:1789760790.108056:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495950464:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.144127:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974085 gen 2 00000008:00010000:1.0:1789760790.144299:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760790.144369:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.144381:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.144403:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760790.144433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495950464:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.144869:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6768, 6768] 00000080:00200000:0.0:1789760790.144886:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.144893:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.144908:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27725824 i_blocks 55305 00000020:00200000:0.0:1789760790.144912:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.144921:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.144955:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27725824, count: 4096 00000080:00200000:0.0:1789760790.144962:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27725824, bytes: 4096 00000080:00200000:0.0:1789760790.144975:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.144985:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.144987:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27725824, 27729920) 0 0 00000080:00200000:0.0:1789760790.144992:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.145007:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6769, 6769] 00020000:00200000:0.0:1789760790.145017:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27725824 chunk: [27725824, 27729920) 27729920, 4096 00020000:00200000:0.0:1789760790.145021:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.145040:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6769, 6769] 00000008:00010000:0.0:1789760790.145058:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.145063:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.145075:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27725824, 27729920) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.145089:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27725824=0x1a71000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.145145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.145148:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27729920 00000020:00010000:0.0:1789760790.145156:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.145160:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.145163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.145202:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.145207:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.145216:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.145440:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.145458:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760790.145723:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 27725824<>27729920 00000100:00100000:1.0:1789760790.145793:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495950592:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.196841:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974086 gen 2 00000008:00010000:1.0:1789760790.196901:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760790.196910:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.196921:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.196945:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760790.196973:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495950592:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760790.197165:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6769, 6769] 00000080:00200000:2.0:1789760790.197208:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.197214:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.197230:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27729920 i_blocks 55305 00000020:00200000:2.0:1789760790.197234:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.197252:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.197296:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27729920, count: 4096 00000080:00200000:2.0:1789760790.197399:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27729920, bytes: 4096 00000080:00200000:2.0:1789760790.197412:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.197421:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.197424:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27729920, 27734016) 0 0 00000080:00200000:2.0:1789760790.197429:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.197447:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6770, 6770] 00020000:00200000:2.0:1789760790.197454:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27729920 chunk: [27729920, 27734016) 27734016, 4096 00020000:00200000:2.0:1789760790.197459:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.197543:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6770, 6770] 00000008:00010000:2.0:1789760790.197562:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.197627:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760790.197640:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27729920, 27734016) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.197653:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27729920=0x1a72000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.197708:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.197712:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27734016 00000020:00010000:2.0:1789760790.197721:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.197725:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.197728:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.197732:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.197735:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.197743:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.197750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.197764:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760790.197937:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266680 - object 0x240000400:4 offset 27729920<>27734016 00000100:00100000:2.0:1789760790.197984:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760790.198001:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495950720:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760790.231842:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760790.231864:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760790.231875:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760790.231901:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760790.231933:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495950720:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.232069:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6770, 6770] 00000080:00200000:0.0:1789760790.232086:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.232092:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.232110:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27734016 i_blocks 55305 00000020:00200000:0.0:1789760790.232113:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.232122:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.232156:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27734016, count: 4096 00000080:00200000:0.0:1789760790.232163:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27734016, bytes: 4096 00000080:00200000:0.0:1789760790.232206:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.232215:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.232218:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27734016, 27738112) 0 0 00000080:00200000:0.0:1789760790.232233:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.232258:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6771, 6771] 00020000:00200000:0.0:1789760790.232266:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27734016 chunk: [27734016, 27738112) 27738112, 4096 00020000:00200000:0.0:1789760790.232270:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.232291:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6771, 6771] 00000008:00010000:0.0:1789760790.232308:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.232314:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.232326:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27734016, 27738112) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.232339:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27734016=0x1a73000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.232394:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.232397:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27738112 00000020:00010000:0.0:1789760790.232406:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.232410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.232413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.232417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.232420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.232428:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.232435:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.232452:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760790.232743:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 27734016<>27738112 00000100:00100000:1.0:1789760790.232824:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495950848:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.289262:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974088 gen 2 00000008:00010000:1.0:1789760790.289329:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760790.289337:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.289347:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.289368:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760790.289398:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495950848:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.289709:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6771, 6771] 00000080:00200000:0.0:1789760790.289726:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.289734:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.289750:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27738112 i_blocks 55305 00000020:00200000:0.0:1789760790.289754:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.289764:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.289795:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27738112, count: 4096 00000080:00200000:0.0:1789760790.289802:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27738112, bytes: 4096 00000080:00200000:0.0:1789760790.289812:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.289821:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.289823:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27738112, 27742208) 0 0 00000080:00200000:0.0:1789760790.289828:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.289845:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6772, 6772] 00020000:00200000:0.0:1789760790.289853:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27738112 chunk: [27738112, 27742208) 27742208, 4096 00020000:00200000:0.0:1789760790.289857:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.289877:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6772, 6772] 00000008:00010000:0.0:1789760790.289895:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.289901:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.289913:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27738112, 27742208) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.289928:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27738112=0x1a74000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.289984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.289987:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27742208 00000020:00010000:0.0:1789760790.289994:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.289998:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.290001:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.290004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.290007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.290015:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.290021:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.290037:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760790.290163:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 27738112<>27742208 00000100:00100000:1.0:1789760790.290447:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495950976:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760790.342527:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760790.342545:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.342558:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.342691:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760790.342723:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495950976:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.342822:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6772, 6772] 00000080:00200000:0.0:1789760790.342840:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.342846:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.342863:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27742208 i_blocks 55305 00000020:00200000:0.0:1789760790.342867:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.342876:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.342908:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27742208, count: 4096 00000080:00200000:0.0:1789760790.342915:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27742208, bytes: 4096 00000080:00200000:0.0:1789760790.342925:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.342933:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.342936:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27742208, 27746304) 0 0 00000080:00200000:0.0:1789760790.342940:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.342955:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6773, 6773] 00020000:00200000:0.0:1789760790.342963:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27742208 chunk: [27742208, 27746304) 27746304, 4096 00020000:00200000:0.0:1789760790.342967:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.342987:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6773, 6773] 00000008:00010000:0.0:1789760790.343005:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.343011:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.343022:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27742208, 27746304) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.343037:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27742208=0x1a75000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.343162:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.343165:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27746304 00000020:00010000:0.0:1789760790.343213:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.343217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.343292:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.343297:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.343300:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.343309:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.343316:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.343334:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760790.343456:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 27742208<>27746304 00000100:00100000:1.0:1789760790.343662:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760790.343683:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495951104:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.381471:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974090 gen 2 00000008:00010000:1.0:1789760790.381534:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760790.381543:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.381553:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.381633:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760790.381664:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495951104:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.381762:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6773, 6773] 00000080:00200000:0.0:1789760790.381781:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.381787:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.381804:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27746304 i_blocks 55305 00000020:00200000:0.0:1789760790.381807:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.381935:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.381970:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27746304, count: 4096 00000080:00200000:0.0:1789760790.381977:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27746304, bytes: 4096 00000080:00200000:0.0:1789760790.381986:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.381995:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.381997:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27746304, 27750400) 0 0 00000080:00200000:0.0:1789760790.382001:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.382019:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6774, 6774] 00020000:00200000:0.0:1789760790.382027:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27746304 chunk: [27746304, 27750400) 27750400, 4096 00020000:00200000:0.0:1789760790.382031:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.382051:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6774, 6774] 00000008:00010000:0.0:1789760790.382069:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.382075:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.382086:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27746304, 27750400) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.382100:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27746304=0x1a76000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.382157:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.382160:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27750400 00000020:00010000:0.0:1789760790.382403:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.382408:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.382411:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.382415:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.382417:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.382426:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.382433:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.382451:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760790.382922:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 27746304<>27750400 00000100:00100000:1.0:1789760790.382987:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495951232:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.416408:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974091 gen 2 00000008:00010000:1.0:1789760790.416457:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760790.416467:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.416478:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.416492:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760790.416520:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495951232:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760790.416694:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6774, 6774] 00000080:00200000:2.0:1789760790.416711:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.416717:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.416733:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27750400 i_blocks 55305 00000020:00200000:2.0:1789760790.416737:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.416746:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.416779:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27750400, count: 4096 00000080:00200000:2.0:1789760790.416787:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27750400, bytes: 4096 00000080:00200000:2.0:1789760790.416796:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.416803:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.416806:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27750400, 27754496) 0 0 00000080:00200000:2.0:1789760790.416924:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.416943:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6775, 6775] 00020000:00200000:2.0:1789760790.416952:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27750400 chunk: [27750400, 27754496) 27754496, 4096 00020000:00200000:2.0:1789760790.416957:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.416978:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6775, 6775] 00000008:00010000:2.0:1789760790.416998:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.417003:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760790.417014:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27750400, 27754496) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.417028:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27750400=0x1a77000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.417084:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.417086:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27754496 00000020:00010000:2.0:1789760790.417095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.417098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.417101:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.417105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.417108:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.417115:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.417121:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.417137:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760790.417479:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265500 - object 0x240000400:4 offset 27750400<>27754496 00000100:00100000:3.0:1789760790.417535:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760790.418368:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495951360:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760790.458576:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760790.458594:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760790.458606:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760790.458629:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760790.458661:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495951360:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760790.458756:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6775, 6775] 00000080:00200000:3.0:1789760790.458772:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.458779:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.458795:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27754496 i_blocks 55305 00000020:00200000:3.0:1789760790.458798:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.458807:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.458841:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27754496, count: 4096 00000080:00200000:3.0:1789760790.458859:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27754496, bytes: 4096 00000080:00200000:3.0:1789760790.458871:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.458880:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760790.458883:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27754496, 27758592) 0 0 00000080:00200000:3.0:1789760790.458887:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.458904:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6776, 6776] 00020000:00200000:3.0:1789760790.458912:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27754496 chunk: [27754496, 27758592) 27758592, 4096 00020000:00200000:3.0:1789760790.458917:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.458937:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6776, 6776] 00000008:00010000:3.0:1789760790.458955:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.458961:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760790.458972:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27754496, 27758592) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.458987:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27754496=0x1a78000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.459056:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.459059:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27758592 00000020:00010000:3.0:1789760790.459068:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.459072:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.459075:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.459079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.459082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.459090:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.459097:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.459113:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760790.459287:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144e00 - object 0x240000400:4 offset 27754496<>27758592 00000100:00100000:3.0:1789760790.459330:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.459348:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495951488:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760790.504994:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974092 gen 2 00000100:00100000:3.0:1789760790.505034:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203144e00 x1876699495951488/t4294974093(4294974093) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760806 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760790.505082:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:3.0:1789760790.505090:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.505101:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.505127:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:3.0:1789760790.505152:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495951488:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760790.505339:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6776, 6776] 00000080:00200000:1.0:1789760790.505356:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.505362:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.505377:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27758592 i_blocks 55305 00000020:00200000:1.0:1789760790.505381:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.505391:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.505426:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27758592, count: 4096 00000080:00200000:1.0:1789760790.505433:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27758592, bytes: 4096 00000080:00200000:1.0:1789760790.505444:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.505452:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760790.505455:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27758592, 27762688) 0 0 00000080:00200000:1.0:1789760790.505459:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.505478:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6777, 6777] 00020000:00200000:1.0:1789760790.505487:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27758592 chunk: [27758592, 27762688) 27762688, 4096 00020000:00200000:1.0:1789760790.505492:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.505515:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6777, 6777] 00000008:00010000:1.0:1789760790.505534:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.505541:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760790.505555:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27758592, 27762688) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.505583:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27758592=0x1a79000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.505648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.505651:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27762688 00000020:00010000:1.0:1789760790.505660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.505663:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.505666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.505669:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.505672:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.505679:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.505688:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.505703:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760790.505834:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27758592<>27762688 00000100:00100000:1.0:1789760790.505886:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760790.505901:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495951616:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.562039:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974094 gen 2 00000008:00010000:1.0:1789760790.562108:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760790.562118:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.562128:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.562151:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760790.562220:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495951616:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760790.562691:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6777, 6777] 00000080:00200000:3.0:1789760790.562710:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.562717:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.562733:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27762688 i_blocks 55305 00000020:00200000:3.0:1789760790.562737:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.562749:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.562785:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27762688, count: 4096 00000080:00200000:3.0:1789760790.562794:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27762688, bytes: 4096 00000080:00200000:3.0:1789760790.562808:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.562934:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760790.562936:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27762688, 27766784) 0 0 00000080:00200000:3.0:1789760790.562941:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.562959:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6778, 6778] 00020000:00200000:3.0:1789760790.562968:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27762688 chunk: [27762688, 27766784) 27766784, 4096 00020000:00200000:3.0:1789760790.562973:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.562995:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6778, 6778] 00000008:00010000:3.0:1789760790.563013:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.563019:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760790.563031:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27762688, 27766784) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.563045:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27762688=0x1a7a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.563107:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.563110:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27766784 00000020:00010000:3.0:1789760790.563118:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.563122:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.563125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.563128:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.563131:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.563139:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.563147:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.563165:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760790.563729:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145500 - object 0x240000400:4 offset 27762688<>27766784 00000100:00100000:2.0:1789760790.563942:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495951744:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760790.602403:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974095 gen 2 00000008:00010000:2.0:1789760790.602459:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760790.602468:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760790.602479:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760790.602502:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760790.602532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495951744:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760790.602691:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6778, 6778] 00000080:00200000:3.0:1789760790.602710:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.602717:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.602732:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27766784 i_blocks 55305 00000020:00200000:3.0:1789760790.602736:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.602746:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.602783:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27766784, count: 4096 00000080:00200000:3.0:1789760790.602791:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27766784, bytes: 4096 00000080:00200000:3.0:1789760790.602801:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.602809:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760790.602812:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27766784, 27770880) 0 0 00000080:00200000:3.0:1789760790.602816:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.602833:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6779, 6779] 00020000:00200000:3.0:1789760790.602842:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27766784 chunk: [27766784, 27770880) 27770880, 4096 00020000:00200000:3.0:1789760790.602846:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.602878:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6779, 6779] 00000008:00010000:3.0:1789760790.602897:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.602903:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760790.602915:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27766784, 27770880) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.602928:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27766784=0x1a7b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.602982:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.602985:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27770880 00000020:00010000:3.0:1789760790.602992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.602996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.602999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.603003:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.603006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.603013:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.603020:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.603036:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760790.603149:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144700 - object 0x240000400:4 offset 27766784<>27770880 00000100:00100000:2.0:1789760790.603394:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495951872:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760790.635416:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974096 gen 2 00000008:00010000:2.0:1789760790.635469:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760790.635480:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760790.635492:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760790.635518:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760790.635554:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495951872:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760790.635685:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6779, 6779] 00000080:00200000:1.0:1789760790.635705:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.635714:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.635731:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27770880 i_blocks 55305 00000020:00200000:1.0:1789760790.635735:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.635746:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.635783:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27770880, count: 4096 00000080:00200000:1.0:1789760790.635790:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27770880, bytes: 4096 00000080:00200000:1.0:1789760790.635800:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.635810:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760790.635813:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27770880, 27774976) 0 0 00000080:00200000:1.0:1789760790.635817:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.635835:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6780, 6780] 00020000:00200000:1.0:1789760790.635843:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27770880 chunk: [27770880, 27774976) 27774976, 4096 00020000:00200000:1.0:1789760790.635849:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.635868:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6780, 6780] 00000008:00010000:1.0:1789760790.635884:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.635891:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760790.635902:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27770880, 27774976) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.635916:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27770880=0x1a7c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.635972:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.635975:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27774976 00000020:00010000:1.0:1789760790.635984:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.635987:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.635991:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.635994:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.635997:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.636004:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.636011:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.636029:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760790.636159:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27770880<>27774976 00000100:00100000:0.0:1789760790.636313:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495952000:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.675153:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974097 gen 2 00000008:00010000:1.0:1789760790.675259:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760790.675268:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.675278:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.675302:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760790.675331:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495952000:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760790.675692:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6780, 6780] 00000080:00200000:2.0:1789760790.675710:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.675717:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.675735:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27774976 i_blocks 55305 00000020:00200000:2.0:1789760790.675740:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.675750:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.675789:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27774976, count: 4096 00000080:00200000:2.0:1789760790.675796:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27774976, bytes: 4096 00000080:00200000:2.0:1789760790.675808:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.675817:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.675820:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27774976, 27779072) 0 0 00000080:00200000:2.0:1789760790.675824:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.675841:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6781, 6781] 00020000:00200000:2.0:1789760790.675862:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27774976 chunk: [27774976, 27779072) 27779072, 4096 00020000:00200000:2.0:1789760790.675867:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.675889:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6781, 6781] 00000008:00010000:2.0:1789760790.675918:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.675925:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760790.675936:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27774976, 27779072) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.675952:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27774976=0x1a7d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.676017:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.676020:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27779072 00000020:00010000:2.0:1789760790.676029:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.676033:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.676037:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.676040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.676043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.676051:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.676059:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.676077:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760790.676262:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 27774976<>27779072 00000100:00100000:2.0:1789760790.676308:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760790.676326:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495952128:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760790.711885:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974098 gen 2 00000008:00010000:2.0:1789760790.711936:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760790.711945:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760790.711957:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760790.711983:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760790.712014:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495952128:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760790.712126:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6781, 6781] 00000080:00200000:1.0:1789760790.712144:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.712151:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.712165:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27779072 i_blocks 55305 00000020:00200000:1.0:1789760790.712198:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.712208:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.712252:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27779072, count: 4096 00000080:00200000:1.0:1789760790.712260:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27779072, bytes: 4096 00000080:00200000:1.0:1789760790.712270:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.712279:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760790.712282:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27779072, 27783168) 0 0 00000080:00200000:1.0:1789760790.712286:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.712302:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6782, 6782] 00020000:00200000:1.0:1789760790.712310:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27779072 chunk: [27779072, 27783168) 27783168, 4096 00020000:00200000:1.0:1789760790.712314:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.712333:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6782, 6782] 00000008:00010000:1.0:1789760790.712348:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.712353:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760790.712362:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27779072, 27783168) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.712375:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27779072=0x1a7e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.712425:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.712428:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27783168 00000020:00010000:1.0:1789760790.712436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.712438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.712441:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.712444:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.712447:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.712453:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.712472:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.712487:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760790.712763:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27779072<>27783168 00000100:00100000:1.0:1789760790.712806:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760790.712823:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495952256:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.755542:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974099 gen 2 00000008:00010000:1.0:1789760790.755593:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:1.0:1789760790.755602:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.755613:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.755735:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:1.0:1789760790.755774:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495952256:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.756468:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6782, 6782] 00000080:00200000:0.0:1789760790.756487:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.756495:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.756513:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27783168 i_blocks 55305 00000020:00200000:0.0:1789760790.756518:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.756533:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.756570:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27783168, count: 4096 00000080:00200000:0.0:1789760790.756578:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27783168, bytes: 4096 00000080:00200000:0.0:1789760790.756588:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.756598:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.756600:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27783168, 27787264) 0 0 00000080:00200000:0.0:1789760790.756605:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.756623:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6783, 6783] 00020000:00200000:0.0:1789760790.756632:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27783168 chunk: [27783168, 27787264) 27787264, 4096 00020000:00200000:0.0:1789760790.756636:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.756657:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6783, 6783] 00000008:00010000:0.0:1789760790.756743:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.756750:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.756761:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27783168, 27787264) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.756776:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27783168=0x1a7f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.756835:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.756838:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27787264 00000020:00010000:0.0:1789760790.756847:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.756851:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.756854:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.756857:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.756861:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.756869:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.756877:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.756899:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760790.757026:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 27783168<>27787264 00000100:00100000:1.0:1789760790.757412:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495952384:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.807920:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974100 gen 2 00000008:00010000:1.0:1789760790.807970:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760790.807979:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.807990:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.808012:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760790.808039:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495952384:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.808658:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6783, 6783] 00000080:00200000:0.0:1789760790.808677:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.808684:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.808700:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27787264 i_blocks 55305 00000020:00200000:0.0:1789760790.808705:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.808714:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.808748:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27787264, count: 4096 00000080:00200000:0.0:1789760790.808755:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27787264, bytes: 4096 00000080:00200000:0.0:1789760790.808767:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.808776:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.808779:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27787264, 27791360) 0 0 00000080:00200000:0.0:1789760790.808783:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.808798:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6784, 6784] 00020000:00200000:0.0:1789760790.808806:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27787264 chunk: [27787264, 27791360) 27791360, 4096 00020000:00200000:0.0:1789760790.808810:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.808830:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6784, 6784] 00000008:00010000:0.0:1789760790.809325:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.809335:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.809350:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27787264, 27791360) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.809366:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27787264=0x1a80000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.809433:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.809436:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27791360 00000020:00010000:0.0:1789760790.809443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.809447:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.809449:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.809453:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.809455:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.809464:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.809472:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.809493:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760790.809709:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 27787264<>27791360 00000100:00100000:1.0:1789760790.810046:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495952512:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.862580:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974101 gen 2 00000008:00010000:1.0:1789760790.862630:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760790.862639:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.862652:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.862673:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760790.862701:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495952512:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.862868:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6784, 6784] 00000080:00200000:0.0:1789760790.862883:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.862890:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.862905:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27791360 i_blocks 55305 00000020:00200000:0.0:1789760790.862909:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.862918:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.862951:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27791360, count: 4096 00000080:00200000:0.0:1789760790.862958:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27791360, bytes: 4096 00000080:00200000:0.0:1789760790.862978:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.862988:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.862990:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27791360, 27795456) 0 0 00000080:00200000:0.0:1789760790.862995:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.863012:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6785, 6785] 00020000:00200000:0.0:1789760790.863020:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27791360 chunk: [27791360, 27795456) 27795456, 4096 00020000:00200000:0.0:1789760790.863025:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.863044:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6785, 6785] 00000008:00010000:0.0:1789760790.863062:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.863068:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.863079:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27791360, 27795456) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.863092:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27791360=0x1a81000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.863151:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.863154:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27795456 00000020:00010000:0.0:1789760790.863163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.863166:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.863202:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.863206:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.863209:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.863227:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.863235:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.863260:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760790.863398:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 27791360<>27795456 00000100:00100000:1.0:1789760790.865897:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.865945:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495952640:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.919548:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974102 gen 2 00000008:00010000:1.0:1789760790.919610:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760790.919619:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.919638:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.919658:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760790.919688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495952640:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.919895:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6785, 6785] 00000080:00200000:0.0:1789760790.919912:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.919918:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.919935:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27795456 i_blocks 55305 00000020:00200000:0.0:1789760790.919938:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.919949:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.919981:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27795456, count: 4096 00000080:00200000:0.0:1789760790.919989:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27795456, bytes: 4096 00000080:00200000:0.0:1789760790.920009:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.920018:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.920020:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27795456, 27799552) 0 0 00000080:00200000:0.0:1789760790.920024:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.920042:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6786, 6786] 00020000:00200000:0.0:1789760790.920051:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27795456 chunk: [27795456, 27799552) 27799552, 4096 00020000:00200000:0.0:1789760790.920056:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.920077:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6786, 6786] 00000008:00010000:0.0:1789760790.920095:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.920100:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.920112:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27795456, 27799552) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.920125:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27795456=0x1a82000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.920222:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.920225:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27799552 00000020:00010000:0.0:1789760790.920234:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.920248:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.920252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.920256:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.920259:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.920266:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.920274:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.920291:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760790.920417:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837480 - object 0x240000400:4 offset 27795456<>27799552 00000100:00100000:1.0:1789760790.920735:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495952768:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760790.954499:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974103 gen 2 00000008:00010000:1.0:1789760790.954546:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760790.954555:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.954690:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.954713:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760790.954744:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495952768:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.955043:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6786, 6786] 00000080:00200000:0.0:1789760790.955060:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.955067:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.955083:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27799552 i_blocks 55305 00000020:00200000:0.0:1789760790.955087:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.955096:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.955128:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27799552, count: 4096 00000080:00200000:0.0:1789760790.955136:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27799552, bytes: 4096 00000080:00200000:0.0:1789760790.955146:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.955154:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.955156:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27799552, 27803648) 0 0 00000080:00200000:0.0:1789760790.955161:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.955537:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6787, 6787] 00020000:00200000:0.0:1789760790.955547:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27799552 chunk: [27799552, 27803648) 27803648, 4096 00020000:00200000:0.0:1789760790.955552:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.955691:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6787, 6787] 00000008:00010000:0.0:1789760790.955710:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.955715:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.955725:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27799552, 27803648) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.955744:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27799552=0x1a83000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.955814:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.955817:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27803648 00000020:00010000:0.0:1789760790.955827:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.955830:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.955833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.955837:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.955839:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.955847:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.955970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.955994:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760790.956134:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834700 - object 0x240000400:4 offset 27799552<>27803648 00000100:00100000:1.0:1789760790.956375:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495952896:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760790.987452:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760790.988908:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974104 gen 2 00000008:00010000:1.0:1789760790.988953:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760790.988962:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.988973:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.988993:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760790.989019:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495952896:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760790.989111:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6787, 6787] 00000080:00200000:0.0:1789760790.989128:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.989134:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.989149:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27803648 i_blocks 55305 00000020:00200000:0.0:1789760790.989153:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.989163:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.989226:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27803648, count: 4096 00000080:00200000:0.0:1789760790.989234:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27803648, bytes: 4096 00000080:00200000:0.0:1789760790.989251:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.989261:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.989263:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27803648, 27807744) 0 0 00000080:00200000:0.0:1789760790.989267:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.989283:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6788, 6788] 00020000:00200000:0.0:1789760790.989291:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27803648 chunk: [27803648, 27807744) 27807744, 4096 00020000:00200000:0.0:1789760790.989296:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.989316:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6788, 6788] 00000008:00010000:0.0:1789760790.989333:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.989339:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760790.989351:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27803648, 27807744) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.989365:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27803648=0x1a84000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.989417:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.989420:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27807744 00000020:00010000:0.0:1789760790.989428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.989431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.989433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.989436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.989439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.989446:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.989454:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.989469:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760790.989590:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835c00 - object 0x240000400:4 offset 27803648<>27807744 00000100:00100000:1.0:1789760790.989761:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495953024:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760791.030617:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760791.030635:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.030647:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.030671:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760791.030706:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495953024:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.030807:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6788, 6788] 00000080:00200000:0.0:1789760791.030824:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.030830:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.030845:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27807744 i_blocks 55305 00000020:00200000:0.0:1789760791.030849:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.030859:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.030888:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27807744, count: 4096 00000080:00200000:0.0:1789760791.030895:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27807744, bytes: 4096 00000080:00200000:0.0:1789760791.030908:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.030916:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.030918:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27807744, 27811840) 0 0 00000080:00200000:0.0:1789760791.030922:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.030938:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6789, 6789] 00020000:00200000:0.0:1789760791.030946:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27807744 chunk: [27807744, 27811840) 27811840, 4096 00020000:00200000:0.0:1789760791.030951:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.030970:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6789, 6789] 00000008:00010000:0.0:1789760791.030986:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.030991:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.031003:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27807744, 27811840) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.031018:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27807744=0x1a85000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.031073:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.031076:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27811840 00000020:00010000:0.0:1789760791.031085:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.031088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.031091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.031094:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.031098:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.031105:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.031113:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.031128:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760791.031418:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834380 - object 0x240000400:4 offset 27807744<>27811840 00000100:00100000:1.0:1789760791.031777:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495953152:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.092361:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974106 gen 2 00000008:00010000:1.0:1789760791.092426:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760791.092434:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.092444:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.092467:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760791.092495:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495953152:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.092695:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6789, 6789] 00000080:00200000:0.0:1789760791.092711:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.092718:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.092734:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27811840 i_blocks 55305 00000020:00200000:0.0:1789760791.092738:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.092747:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.092782:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27811840, count: 4096 00000080:00200000:0.0:1789760791.092789:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27811840, bytes: 4096 00000080:00200000:0.0:1789760791.092799:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.092809:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.092811:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27811840, 27815936) 0 0 00000080:00200000:0.0:1789760791.092816:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.092833:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6790, 6790] 00020000:00200000:0.0:1789760791.092842:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27811840 chunk: [27811840, 27815936) 27815936, 4096 00020000:00200000:0.0:1789760791.092847:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.092866:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6790, 6790] 00000008:00010000:0.0:1789760791.092885:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.092891:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.092902:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27811840, 27815936) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.092916:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27811840=0x1a86000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.092975:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.092978:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27815936 00000020:00010000:0.0:1789760791.092986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.092990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.092992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.092996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.092999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.093005:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.093012:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.093028:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760791.093304:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835180 - object 0x240000400:4 offset 27811840<>27815936 00000100:00100000:1.0:1789760791.093408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495953280:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760791.153760:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:1.0:1789760791.153777:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.153789:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.153809:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:1.0:1789760791.153842:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495953280:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.154130:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6790, 6790] 00000080:00200000:0.0:1789760791.154149:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.154154:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.154195:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27815936 i_blocks 55305 00000020:00200000:0.0:1789760791.154199:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.154208:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.154253:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27815936, count: 4096 00000080:00200000:0.0:1789760791.154260:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27815936, bytes: 4096 00000080:00200000:0.0:1789760791.154269:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.154278:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.154280:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27815936, 27820032) 0 0 00000080:00200000:0.0:1789760791.154284:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.154300:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6791, 6791] 00020000:00200000:0.0:1789760791.154308:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27815936 chunk: [27815936, 27820032) 27820032, 4096 00020000:00200000:0.0:1789760791.154313:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.154339:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6791, 6791] 00000008:00010000:0.0:1789760791.154421:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.154427:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.154438:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27815936, 27820032) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.154453:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27815936=0x1a87000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.154511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.154514:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27820032 00000020:00010000:0.0:1789760791.154522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.154526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.154529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.154533:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.154536:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.154544:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.154555:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.154572:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760791.154692:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834a80 - object 0x240000400:4 offset 27815936<>27820032 00000100:00100000:1.0:1789760791.155117:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495953408:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.192736:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974108 gen 2 00000008:00010000:1.0:1789760791.192801:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:1.0:1789760791.192809:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.192819:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.193065:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:1.0:1789760791.193096:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495953408:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.193191:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6791, 6791] 00000080:00200000:0.0:1789760791.193209:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.193290:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.193309:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27820032 i_blocks 55305 00000020:00200000:0.0:1789760791.193312:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.193322:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.193357:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27820032, count: 4096 00000080:00200000:0.0:1789760791.193365:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27820032, bytes: 4096 00000080:00200000:0.0:1789760791.193376:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.193384:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.193387:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27820032, 27824128) 0 0 00000080:00200000:0.0:1789760791.193391:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.193408:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6792, 6792] 00020000:00200000:0.0:1789760791.193417:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27820032 chunk: [27820032, 27824128) 27824128, 4096 00020000:00200000:0.0:1789760791.193422:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.193444:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6792, 6792] 00000008:00010000:0.0:1789760791.193463:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.193469:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.193481:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27820032, 27824128) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.193496:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27820032=0x1a88000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.193556:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.193625:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27824128 00000020:00010000:0.0:1789760791.193634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.193638:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.193641:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.193645:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.193648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.193655:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.193663:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.193681:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760791.193807:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836a00 - object 0x240000400:4 offset 27820032<>27824128 00000100:00100000:1.0:1789760791.193877:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495953536:192.168.203.109@tcp:4:dd.0 00000008:00010000:1.0:1789760791.242875:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760791.242892:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.242906:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.242939:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760791.242978:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495953536:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.243122:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6792, 6792] 00000080:00200000:0.0:1789760791.243139:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.243146:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.243163:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27824128 i_blocks 55305 00000020:00200000:0.0:1789760791.243166:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.243199:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.243233:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27824128, count: 4096 00000080:00200000:0.0:1789760791.247722:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27824128, bytes: 4096 00000080:00200000:0.0:1789760791.247746:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.247758:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.247761:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27824128, 27828224) 0 0 00000080:00200000:0.0:1789760791.247767:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.247799:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6793, 6793] 00020000:00200000:0.0:1789760791.247808:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27824128 chunk: [27824128, 27828224) 27828224, 4096 00020000:00200000:0.0:1789760791.247814:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.247838:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6793, 6793] 00000008:00010000:0.0:1789760791.247861:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.247866:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.247878:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27824128, 27828224) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.247894:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27824128=0x1a89000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.247980:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.247984:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27828224 00000020:00010000:0.0:1789760791.247993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.247996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.248000:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.248004:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.248007:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.248021:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.248031:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.248066:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760791.248250:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835880 - object 0x240000400:4 offset 27824128<>27828224 00000100:00100000:1.0:1789760791.248958:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495953664:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.282224:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974110 gen 2 00000008:00010000:1.0:1789760791.282314:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760791.282324:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.282336:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.282349:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760791.282392:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495953664:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.282838:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6793, 6793] 00000080:00200000:0.0:1789760791.282855:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.282862:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.282878:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27828224 i_blocks 55305 00000020:00200000:0.0:1789760791.282882:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.282894:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.282930:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27828224, count: 4096 00000080:00200000:0.0:1789760791.282938:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27828224, bytes: 4096 00000080:00200000:0.0:1789760791.282948:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.282958:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.282960:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27828224, 27832320) 0 0 00000080:00200000:0.0:1789760791.282965:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.282983:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6794, 6794] 00020000:00200000:0.0:1789760791.282991:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27828224 chunk: [27828224, 27832320) 27832320, 4096 00020000:00200000:0.0:1789760791.282996:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.283017:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6794, 6794] 00000008:00010000:0.0:1789760791.283036:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.283042:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.283053:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27828224, 27832320) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.283067:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27828224=0x1a8a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.283125:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.283128:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27832320 00000020:00010000:0.0:1789760791.283136:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.283140:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.283143:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.283148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.283150:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.283158:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.283165:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.283225:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760791.283371:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834e00 - object 0x240000400:4 offset 27828224<>27832320 00000100:00100000:1.0:1789760791.284774:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760791.284801:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495953792:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.328244:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974111 gen 2 00000008:00010000:1.0:1789760791.328297:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760791.328305:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.328316:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.328327:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760791.328357:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495953792:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.328465:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6794, 6794] 00000080:00200000:0.0:1789760791.328483:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.328490:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.328506:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27832320 i_blocks 55305 00000020:00200000:0.0:1789760791.328510:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.328895:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.328932:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27832320, count: 4096 00000080:00200000:0.0:1789760791.328939:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27832320, bytes: 4096 00000080:00200000:0.0:1789760791.328960:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.328969:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.328971:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27832320, 27836416) 0 0 00000080:00200000:0.0:1789760791.328976:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.328991:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6795, 6795] 00020000:00200000:0.0:1789760791.328999:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27832320 chunk: [27832320, 27836416) 27836416, 4096 00020000:00200000:0.0:1789760791.329004:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.329159:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6795, 6795] 00000008:00010000:0.0:1789760791.329256:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.329262:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.329272:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27832320, 27836416) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.329287:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27832320=0x1a8b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.329340:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.329342:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27836416 00000020:00010000:0.0:1789760791.329350:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.329354:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.329357:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.329360:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.329362:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.329371:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.329379:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.329393:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:0.0:1789760791.329508:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836680 - object 0x240000400:4 offset 27832320<>27836416 00000100:00100000:1.0:1789760791.329651:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.329672:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495953920:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.361520:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974112 gen 2 00000008:00010000:1.0:1789760791.361677:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760791.361686:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.361696:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.361764:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760791.361806:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495953920:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.363697:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6795, 6795] 00000080:00200000:0.0:1789760791.363722:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.363729:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.363747:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27836416 i_blocks 55305 00000020:00200000:0.0:1789760791.363752:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.363765:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.363868:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27836416, count: 4096 00000080:00200000:0.0:1789760791.363879:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27836416, bytes: 4096 00000080:00200000:0.0:1789760791.363897:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.363908:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.363910:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27836416, 27840512) 0 0 00000080:00200000:0.0:1789760791.363915:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.363931:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6796, 6796] 00020000:00200000:0.0:1789760791.363940:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27836416 chunk: [27836416, 27840512) 27840512, 4096 00020000:00200000:0.0:1789760791.363945:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.363967:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6796, 6796] 00000008:00010000:0.0:1789760791.363987:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.363993:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.364005:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27836416, 27840512) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.364020:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27836416=0x1a8c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.364092:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.364095:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27840512 00000020:00010000:0.0:1789760791.364104:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.364108:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.364111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.364114:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.364117:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.364125:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.364132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.364160:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760791.364508:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211834000 - object 0x240000400:4 offset 27836416<>27840512 00000100:00100000:1.0:1789760791.364614:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495954048:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.411993:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974113 gen 2 00000008:00010000:1.0:1789760791.412045:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760791.412054:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.412066:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.412129:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760791.412159:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495954048:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.413695:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6796, 6796] 00000080:00200000:0.0:1789760791.413720:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.413727:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.413745:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27840512 i_blocks 55305 00000020:00200000:0.0:1789760791.413750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.413761:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.413982:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27840512, count: 4096 00000080:00200000:0.0:1789760791.413991:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27840512, bytes: 4096 00000080:00200000:0.0:1789760791.414005:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.414015:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.414017:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27840512, 27844608) 0 0 00000080:00200000:0.0:1789760791.414022:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.414039:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6797, 6797] 00020000:00200000:0.0:1789760791.414046:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27840512 chunk: [27840512, 27844608) 27844608, 4096 00020000:00200000:0.0:1789760791.414051:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.414073:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6797, 6797] 00000008:00010000:0.0:1789760791.414091:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.414097:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.414109:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27840512, 27844608) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.414124:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27840512=0x1a8d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.414750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.414756:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27844608 00000020:00010000:0.0:1789760791.414766:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.414769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.414772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.414776:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.414779:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.414900:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.414910:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.414941:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760791.415074:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 27840512<>27844608 00000100:00100000:1.0:1789760791.415514:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495954176:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.460461:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974114 gen 2 00000008:00010000:1.0:1789760791.460512:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760791.460520:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.460532:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.460636:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760791.460664:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495954176:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.460783:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6797, 6797] 00000080:00200000:0.0:1789760791.460799:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.460806:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.460821:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27844608 i_blocks 55305 00000020:00200000:0.0:1789760791.460825:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.460833:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.460865:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27844608, count: 4096 00000080:00200000:0.0:1789760791.460872:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27844608, bytes: 4096 00000080:00200000:0.0:1789760791.460886:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.460896:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.460898:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27844608, 27848704) 0 0 00000080:00200000:0.0:1789760791.460903:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.460919:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6798, 6798] 00020000:00200000:0.0:1789760791.460928:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27844608 chunk: [27844608, 27848704) 27848704, 4096 00020000:00200000:0.0:1789760791.460933:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.460951:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6798, 6798] 00000008:00010000:0.0:1789760791.460968:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.460974:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.460987:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27844608, 27848704) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.461074:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27844608=0x1a8e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.461129:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.461132:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27848704 00000020:00010000:0.0:1789760791.461140:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.461144:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.461147:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.461151:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.461154:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.461160:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.461200:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.461215:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760791.461363:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 27844608<>27848704 00000100:00100000:1.0:1789760791.461471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495954304:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.530875:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974115 gen 2 00000008:00010000:1.0:1789760791.530997:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760791.531005:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.531017:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.531038:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760791.531068:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495954304:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.531203:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6798, 6798] 00000080:00200000:0.0:1789760791.531219:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.531225:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.531249:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27848704 i_blocks 55305 00000020:00200000:0.0:1789760791.531252:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.531263:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.531295:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27848704, count: 4096 00000080:00200000:0.0:1789760791.531301:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27848704, bytes: 4096 00000080:00200000:0.0:1789760791.531310:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.531319:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.531321:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27848704, 27852800) 0 0 00000080:00200000:0.0:1789760791.531325:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.531342:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6799, 6799] 00020000:00200000:0.0:1789760791.531350:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27848704 chunk: [27848704, 27852800) 27852800, 4096 00020000:00200000:0.0:1789760791.531356:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.531376:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6799, 6799] 00000008:00010000:0.0:1789760791.531394:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.531400:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.531411:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27848704, 27852800) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.531424:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27848704=0x1a8f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.531480:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.531483:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27852800 00000020:00010000:0.0:1789760791.531491:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.531494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.531497:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.531500:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.531503:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.531510:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.531517:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.531534:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760791.532058:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 27848704<>27852800 00000100:00100000:1.0:1789760791.532137:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495954432:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.576258:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974116 gen 2 00000008:00010000:1.0:1789760791.576310:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760791.576318:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.576329:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.576353:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760791.576384:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495954432:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.576505:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6799, 6799] 00000080:00200000:0.0:1789760791.576533:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.576540:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.576557:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27852800 i_blocks 55305 00000020:00200000:0.0:1789760791.576561:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.576569:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.576603:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27852800, count: 4096 00000080:00200000:0.0:1789760791.576611:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27852800, bytes: 4096 00000080:00200000:0.0:1789760791.576622:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.576632:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.576635:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27852800, 27856896) 0 0 00000080:00200000:0.0:1789760791.576639:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.576655:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6800, 6800] 00020000:00200000:0.0:1789760791.576664:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27852800 chunk: [27852800, 27856896) 27856896, 4096 00020000:00200000:0.0:1789760791.576669:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.576689:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6800, 6800] 00000008:00010000:0.0:1789760791.576707:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.576713:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.576725:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27852800, 27856896) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.576740:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27852800=0x1a90000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.576800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.576803:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27856896 00000020:00010000:0.0:1789760791.576811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.576815:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.576818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.576821:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.576825:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.576833:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.576839:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.576857:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ef00 00000008:00100000:0.0:1789760791.576976:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27852800<>27856896 00000100:00100000:1.0:1789760791.577306:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495954560:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.608705:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974117 gen 2 00000008:00010000:1.0:1789760791.608758:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ef00 00000020:00200000:1.0:1789760791.608767:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.608777:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.608798:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ef00 00000100:00100000:1.0:1789760791.608831:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495954560:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.608939:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6800, 6800] 00000080:00200000:0.0:1789760791.608957:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.608963:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.608979:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27856896 i_blocks 55305 00000020:00200000:0.0:1789760791.608983:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.608993:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.609026:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27856896, count: 4096 00000080:00200000:0.0:1789760791.609033:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27856896, bytes: 4096 00000080:00200000:0.0:1789760791.609042:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.609052:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.609055:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27856896, 27860992) 0 0 00000080:00200000:0.0:1789760791.609059:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.609077:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6801, 6801] 00020000:00200000:0.0:1789760791.609086:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27856896 chunk: [27856896, 27860992) 27860992, 4096 00020000:00200000:0.0:1789760791.609091:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.609111:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6801, 6801] 00000008:00010000:0.0:1789760791.609127:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.609133:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.609145:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27856896, 27860992) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.609159:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27856896=0x1a91000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.609392:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.609395:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27860992 00000020:00010000:0.0:1789760791.609404:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.609407:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.609410:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.609414:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.609416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.609423:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.609431:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.609448:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e240 00000008:00100000:0.0:1789760791.609584:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 27856896<>27860992 00000100:00100000:1.0:1789760791.609785:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495954688:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.638710:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974118 gen 2 00000008:00010000:1.0:1789760791.638762:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e240 00000020:00200000:1.0:1789760791.638769:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.638781:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.638801:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e240 00000100:00100000:1.0:1789760791.638848:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495954688:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.639103:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6801, 6801] 00000080:00200000:0.0:1789760791.639120:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.639126:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.639141:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27860992 i_blocks 55305 00000020:00200000:0.0:1789760791.639144:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.639154:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.639216:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27860992, count: 4096 00000080:00200000:0.0:1789760791.639223:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27860992, bytes: 4096 00000080:00200000:0.0:1789760791.639247:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.639257:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.639259:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27860992, 27865088) 0 0 00000080:00200000:0.0:1789760791.639265:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.639281:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6802, 6802] 00020000:00200000:0.0:1789760791.639290:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27860992 chunk: [27860992, 27865088) 27865088, 4096 00020000:00200000:0.0:1789760791.639295:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.639313:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6802, 6802] 00000008:00010000:0.0:1789760791.639331:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.639336:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.639347:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27860992, 27865088) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.639360:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27860992=0x1a92000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.639409:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.639411:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27865088 00000020:00010000:0.0:1789760791.639418:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.639422:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.639425:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.639428:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.639431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.639438:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.639444:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.639459:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e3c0 00000008:00100000:0.0:1789760791.639583:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 27860992<>27865088 00000100:00100000:1.0:1789760791.639760:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495954816:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.669444:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974119 gen 2 00000008:00010000:1.0:1789760791.669495:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e3c0 00000020:00200000:1.0:1789760791.669505:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.669515:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.669540:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e3c0 00000100:00100000:1.0:1789760791.669573:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495954816:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.669693:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6802, 6802] 00000080:00200000:0.0:1789760791.669712:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.669718:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.669735:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27865088 i_blocks 55305 00000020:00200000:0.0:1789760791.669738:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.669748:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.669783:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27865088, count: 4096 00000080:00200000:0.0:1789760791.669791:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27865088, bytes: 4096 00000080:00200000:0.0:1789760791.669802:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.669811:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.669814:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27865088, 27869184) 0 0 00000080:00200000:0.0:1789760791.669819:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.669837:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6803, 6803] 00020000:00200000:0.0:1789760791.669845:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27865088 chunk: [27865088, 27869184) 27869184, 4096 00020000:00200000:0.0:1789760791.669850:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.669870:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6803, 6803] 00000008:00010000:0.0:1789760791.669888:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.669894:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.669907:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27865088, 27869184) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.669922:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27865088=0x1a93000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.669984:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.669987:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27869184 00000020:00010000:0.0:1789760791.669996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.669999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.670002:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.670006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.670009:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.670016:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.670024:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.670041:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845eb40 00000008:00100000:0.0:1789760791.670166:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 27865088<>27869184 00000100:00100000:1.0:1789760791.670427:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495954944:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.704611:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974120 gen 2 00000008:00010000:1.0:1789760791.704673:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845eb40 00000020:00200000:1.0:1789760791.704681:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.704697:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.704719:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845eb40 00000100:00100000:1.0:1789760791.704753:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495954944:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.705058:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6803, 6803] 00000080:00200000:0.0:1789760791.705078:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.705085:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.705104:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27869184 i_blocks 55305 00000020:00200000:0.0:1789760791.705110:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.705122:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.705166:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27869184, count: 4096 00000080:00200000:0.0:1789760791.705219:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27869184, bytes: 4096 00000080:00200000:0.0:1789760791.705239:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.705251:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.705254:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27869184, 27873280) 0 0 00000080:00200000:0.0:1789760791.705258:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.705279:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6804, 6804] 00020000:00200000:0.0:1789760791.705287:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27869184 chunk: [27869184, 27873280) 27873280, 4096 00020000:00200000:0.0:1789760791.705303:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.705450:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6804, 6804] 00000008:00010000:0.0:1789760791.705472:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.705478:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.705490:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27869184, 27873280) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.705504:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27869184=0x1a94000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.705557:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.705559:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27873280 00000020:00010000:0.0:1789760791.705567:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.705571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.705574:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.705578:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.705581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.705589:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.705601:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.705718:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e480 00000008:00100000:1.0:1789760791.705857:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27869184<>27873280 00000100:00100000:1.0:1789760791.705909:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495955072:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.769215:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974121 gen 2 00000008:00010000:1.0:1789760791.769655:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e480 00000020:00200000:1.0:1789760791.769664:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.769680:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.769711:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e480 00000100:00100000:1.0:1789760791.769740:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495955072:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.770612:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6804, 6804] 00000080:00200000:0.0:1789760791.770633:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.770640:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.770657:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27873280 i_blocks 55305 00000020:00200000:0.0:1789760791.770661:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.770672:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.770707:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27873280, count: 4096 00000080:00200000:0.0:1789760791.770715:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27873280, bytes: 4096 00000080:00200000:0.0:1789760791.770730:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.770741:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.770743:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27873280, 27877376) 0 0 00000080:00200000:0.0:1789760791.770748:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.770842:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6805, 6805] 00020000:00200000:0.0:1789760791.770850:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27873280 chunk: [27873280, 27877376) 27877376, 4096 00020000:00200000:0.0:1789760791.770855:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.770876:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6805, 6805] 00000008:00010000:0.0:1789760791.770895:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.770901:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.770913:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27873280, 27877376) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.770928:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27873280=0x1a95000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.770990:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.770993:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27877376 00000020:00010000:0.0:1789760791.771001:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.771006:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.771009:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.771012:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.771015:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.771023:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.771030:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.772030:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ed80 00000008:00100000:0.0:1789760791.772833:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836300 - object 0x240000400:4 offset 27873280<>27877376 00000100:00100000:1.0:1789760791.772934:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495955200:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.834264:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974122 gen 2 00000008:00010000:1.0:1789760791.834313:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ed80 00000020:00200000:1.0:1789760791.834331:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.834342:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.834360:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ed80 00000100:00100000:1.0:1789760791.834388:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495955200:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.834602:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6805, 6805] 00000080:00200000:0.0:1789760791.834619:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.834624:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.834639:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27877376 i_blocks 55305 00000020:00200000:0.0:1789760791.834642:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.834650:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.834679:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27877376, count: 4096 00000080:00200000:0.0:1789760791.834686:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27877376, bytes: 4096 00000080:00200000:0.0:1789760791.834695:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.834702:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.834705:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27877376, 27881472) 0 0 00000080:00200000:0.0:1789760791.834709:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.834725:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6806, 6806] 00020000:00200000:0.0:1789760791.834733:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27877376 chunk: [27877376, 27881472) 27881472, 4096 00020000:00200000:0.0:1789760791.834737:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.834757:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6806, 6806] 00000008:00010000:0.0:1789760791.834773:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.834779:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.834789:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27877376, 27881472) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.834802:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27877376=0x1a96000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.834853:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.834856:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27881472 00000020:00010000:0.0:1789760791.834864:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.834867:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.834870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.834873:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.834875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.834882:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.834888:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.834903:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e180 00000008:00100000:0.0:1789760791.835141:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837100 - object 0x240000400:4 offset 27877376<>27881472 00000100:00100000:1.0:1789760791.836108:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495955328:192.168.203.109@tcp:4:dd.0 00000100:00100000:1.0:1789760791.867138:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974123 gen 2 00000008:00010000:1.0:1789760791.867361:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e180 00000020:00200000:1.0:1789760791.867370:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.867381:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.867475:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e180 00000100:00100000:1.0:1789760791.867506:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495955328:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760791.867868:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6806, 6806] 00000080:00200000:2.0:1789760791.867885:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.867891:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.867909:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27881472 i_blocks 55305 00000020:00200000:2.0:1789760791.867913:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.867923:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.868026:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27881472, count: 4096 00000080:00200000:2.0:1789760791.868035:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27881472, bytes: 4096 00000080:00200000:2.0:1789760791.868046:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.868055:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.868057:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27881472, 27885568) 0 0 00000080:00200000:2.0:1789760791.868062:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.868079:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6807, 6807] 00020000:00200000:2.0:1789760791.868088:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27881472 chunk: [27881472, 27885568) 27885568, 4096 00020000:00200000:2.0:1789760791.868093:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.868113:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6807, 6807] 00000008:00010000:2.0:1789760791.868131:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.868138:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760791.868149:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27881472, 27885568) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.868163:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27881472=0x1a97000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.868404:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.868408:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27885568 00000020:00010000:2.0:1789760791.868488:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.868493:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.868498:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.868501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.868504:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.868514:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.868521:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.868545:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760791.868933:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 27881472<>27885568 00000100:00100000:3.0:1789760791.869098:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495955456:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760791.912749:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760791.912765:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.912778:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.912803:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760791.912837:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495955456:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760791.912943:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6807, 6807] 00000080:00200000:2.0:1789760791.912960:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.912967:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.912985:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27885568 i_blocks 55305 00000020:00200000:2.0:1789760791.913062:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.913073:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.913109:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27885568, count: 4096 00000080:00200000:2.0:1789760791.913118:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27885568, bytes: 4096 00000080:00200000:2.0:1789760791.913128:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.913137:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.913139:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27885568, 27889664) 0 0 00000080:00200000:2.0:1789760791.913144:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.913161:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6808, 6808] 00020000:00200000:2.0:1789760791.913257:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27885568 chunk: [27885568, 27889664) 27889664, 4096 00020000:00200000:2.0:1789760791.913263:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.913285:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6808, 6808] 00000008:00010000:2.0:1789760791.913303:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.913308:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760791.913321:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27885568, 27889664) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.913336:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27885568=0x1a98000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.913397:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.913400:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27889664 00000020:00010000:2.0:1789760791.913409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.913413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.913416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.913420:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.913423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.913431:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.913437:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.913455:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760791.913585:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 27885568<>27889664 00000100:00100000:3.0:1789760791.913656:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495955584:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760791.973888:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974125 gen 2 00000008:00010000:2.0:1789760791.973950:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:2.0:1789760791.973959:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760791.973968:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760791.975055:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:2.0:1789760791.975087:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495955584:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760791.986328:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6808, 6808] 00000080:00200000:0.0:1789760791.986359:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.986367:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.986387:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27889664 i_blocks 55305 00000020:00200000:0.0:1789760791.986393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.986410:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.986442:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27889664, count: 4096 00000080:00200000:0.0:1789760791.986450:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27889664, bytes: 4096 00000080:00200000:0.0:1789760791.986465:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.986474:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.986477:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27889664, 27893760) 0 0 00000080:00200000:0.0:1789760791.986481:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.986498:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6809, 6809] 00020000:00200000:0.0:1789760791.986506:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27889664 chunk: [27889664, 27893760) 27893760, 4096 00020000:00200000:0.0:1789760791.986511:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.986533:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6809, 6809] 00000008:00010000:0.0:1789760791.986552:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.986558:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760791.986571:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27889664, 27893760) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.986597:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27889664=0x1a99000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.986676:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.986679:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27893760 00000020:00010000:0.0:1789760791.986687:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.986690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.986693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.986696:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.986699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.986707:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.986714:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.986742:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845e0c0 00000008:00100000:0.0:1789760791.986876:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837b80 - object 0x240000400:4 offset 27889664<>27893760 00000100:00100000:1.0:1789760791.987398:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495955712:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760792.012887:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760792.032684:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974126 gen 2 00000008:00010000:1.0:1789760792.032739:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845e0c0 00000020:00200000:1.0:1789760792.032748:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.032770:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.032794:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845e0c0 00000100:00100000:1.0:1789760792.032825:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495955712:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760792.032995:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6809, 6809] 00000080:00200000:0.0:1789760792.033013:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.033020:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.033035:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27893760 i_blocks 55305 00000020:00200000:0.0:1789760792.033039:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.033050:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.033084:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27893760, count: 4096 00000080:00200000:0.0:1789760792.033091:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27893760, bytes: 4096 00000080:00200000:0.0:1789760792.033103:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.033112:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.033115:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27893760, 27897856) 0 0 00000080:00200000:0.0:1789760792.033119:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.033136:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6810, 6810] 00020000:00200000:0.0:1789760792.033144:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27893760 chunk: [27893760, 27897856) 27897856, 4096 00020000:00200000:0.0:1789760792.033150:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.033547:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6810, 6810] 00000008:00010000:0.0:1789760792.033607:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.033614:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760792.033627:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27893760, 27897856) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.033644:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27893760=0x1a9a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.033701:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.033704:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27897856 00000020:00010000:0.0:1789760792.033713:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.033717:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.033720:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.033724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.033728:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.033735:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.033743:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.033756:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:0.0:1789760792.033944:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27893760<>27897856 00000100:00100000:1.0:1789760792.034154:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760792.034361:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495955840:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.069249:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974127 gen 2 00000008:00010000:0.0:1789760792.069305:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:0.0:1789760792.069314:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.069324:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.069353:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:0.0:1789760792.069382:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495955840:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.069483:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6810, 6810] 00000080:00200000:1.0:1789760792.069499:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.069505:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.069522:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27897856 i_blocks 55305 00000020:00200000:1.0:1789760792.069526:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.069537:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.069586:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27897856, count: 4096 00000080:00200000:1.0:1789760792.069594:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27897856, bytes: 4096 00000080:00200000:1.0:1789760792.069605:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.069613:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.069616:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27897856, 27901952) 0 0 00000080:00200000:1.0:1789760792.069621:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.069638:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6811, 6811] 00020000:00200000:1.0:1789760792.069647:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27897856 chunk: [27897856, 27901952) 27901952, 4096 00020000:00200000:1.0:1789760792.069652:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.069673:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6811, 6811] 00000008:00010000:1.0:1789760792.069691:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.069696:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.069707:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27897856, 27901952) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.069721:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27897856=0x1a9b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.069778:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.069780:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27901952 00000020:00010000:1.0:1789760792.069789:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.069793:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.069796:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.069799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.069803:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.069810:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.069817:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.069831:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760792.069956:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27897856<>27901952 00000100:00100000:0.0:1789760792.070065:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760792.070083:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495955968:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.106262:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974128 gen 2 00000008:00010000:0.0:1789760792.106311:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760792.106320:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.106330:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.106358:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760792.106386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495955968:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.106565:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6811, 6811] 00000080:00200000:1.0:1789760792.106584:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.106590:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.106606:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27901952 i_blocks 55305 00000020:00200000:1.0:1789760792.106610:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.106619:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.106653:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27901952, count: 4096 00000080:00200000:1.0:1789760792.106666:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27901952, bytes: 4096 00000080:00200000:1.0:1789760792.106677:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.106685:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.106688:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27901952, 27906048) 0 0 00000080:00200000:1.0:1789760792.106692:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.106708:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6812, 6812] 00020000:00200000:1.0:1789760792.106716:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27901952 chunk: [27901952, 27906048) 27906048, 4096 00020000:00200000:1.0:1789760792.106720:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.106739:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6812, 6812] 00000008:00010000:1.0:1789760792.106756:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.106763:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.106775:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27901952, 27906048) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.106788:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27901952=0x1a9c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.106847:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.106850:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27906048 00000020:00010000:1.0:1789760792.106858:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.106862:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.106865:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.106870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.106873:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.106881:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.106888:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.106904:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760792.107030:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 27901952<>27906048 00000100:00100000:0.0:1789760792.107208:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760792.107238:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956096:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760792.151622:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760792.151636:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.151646:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.151664:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760792.151692:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956096:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.151841:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6812, 6812] 00000080:00200000:1.0:1789760792.151857:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.151863:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.151878:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27906048 i_blocks 55305 00000020:00200000:1.0:1789760792.151882:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.151890:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.151921:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27906048, count: 4096 00000080:00200000:1.0:1789760792.151928:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27906048, bytes: 4096 00000080:00200000:1.0:1789760792.151936:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.151944:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.151947:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27906048, 27910144) 0 0 00000080:00200000:1.0:1789760792.151951:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.151966:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6813, 6813] 00020000:00200000:1.0:1789760792.151973:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27906048 chunk: [27906048, 27910144) 27910144, 4096 00020000:00200000:1.0:1789760792.151977:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.151995:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6813, 6813] 00000008:00010000:1.0:1789760792.152011:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.152017:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.152027:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27906048, 27910144) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.152041:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27906048=0x1a9d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.152098:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.152101:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27910144 00000020:00010000:1.0:1789760792.152109:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.152113:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.152116:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.152119:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.152123:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.152131:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.152138:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.152153:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760792.152325:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 27906048<>27910144 00000100:00100000:0.0:1789760792.152781:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760792.152804:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495956224:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.188259:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974129 gen 2 00000100:00100000:0.0:1789760792.188451:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504dc00 x1876699495956224/t4294974130(4294974130) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760808 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760792.188500:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760792.188506:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.188515:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.188535:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760792.188565:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495956224:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.188697:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6813, 6813] 00000080:00200000:1.0:1789760792.188713:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.188720:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.188735:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27910144 i_blocks 55305 00000020:00200000:1.0:1789760792.188739:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.188747:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.188780:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27910144, count: 4096 00000080:00200000:1.0:1789760792.188787:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27910144, bytes: 4096 00000080:00200000:1.0:1789760792.188798:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.188806:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.188808:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27910144, 27914240) 0 0 00000080:00200000:1.0:1789760792.188812:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.188830:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6814, 6814] 00020000:00200000:1.0:1789760792.188838:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27910144 chunk: [27910144, 27914240) 27914240, 4096 00020000:00200000:1.0:1789760792.188843:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.188862:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6814, 6814] 00000008:00010000:1.0:1789760792.188881:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.188886:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.188898:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27910144, 27914240) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.188913:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27910144=0x1a9e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.188975:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.188978:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27914240 00000020:00010000:1.0:1789760792.188986:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.188990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.188993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.188996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.188999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.189006:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.189013:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.189028:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760792.189157:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 27910144<>27914240 00000100:00100000:0.0:1789760792.189491:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760792.189510:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956352:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.228538:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974131 gen 2 00000008:00010000:0.0:1789760792.228671:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760792.228680:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.228690:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.228774:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760792.228805:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956352:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.228940:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6814, 6814] 00000080:00200000:1.0:1789760792.228957:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.228963:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.228979:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27914240 i_blocks 55305 00000020:00200000:1.0:1789760792.228982:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.229053:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.229087:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27914240, count: 4096 00000080:00200000:1.0:1789760792.229094:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27914240, bytes: 4096 00000080:00200000:1.0:1789760792.229107:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.229117:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.229119:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27914240, 27918336) 0 0 00000080:00200000:1.0:1789760792.229124:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.229141:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6815, 6815] 00020000:00200000:1.0:1789760792.229150:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27914240 chunk: [27914240, 27918336) 27918336, 4096 00020000:00200000:1.0:1789760792.229154:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.229200:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6815, 6815] 00000008:00010000:1.0:1789760792.229377:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.229383:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.229396:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27914240, 27918336) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.229411:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27914240=0x1a9f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.229469:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.229471:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27918336 00000020:00010000:1.0:1789760792.229480:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.229484:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.229487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.229490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.229495:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.229503:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.229511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.229530:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760792.229733:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 27914240<>27918336 00000100:00100000:0.0:1789760792.229810:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956480:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.275503:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974132 gen 2 00000008:00010000:0.0:1789760792.275556:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760792.275566:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.275576:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.275916:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760792.275945:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956480:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.276150:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6815, 6815] 00000080:00200000:1.0:1789760792.276190:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.276197:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.276213:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27918336 i_blocks 55305 00000020:00200000:1.0:1789760792.276217:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.276236:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.276266:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27918336, count: 4096 00000080:00200000:1.0:1789760792.276273:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27918336, bytes: 4096 00000080:00200000:1.0:1789760792.276283:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.276356:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.276359:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27918336, 27922432) 0 0 00000080:00200000:1.0:1789760792.276363:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.276381:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6816, 6816] 00020000:00200000:1.0:1789760792.276389:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27918336 chunk: [27918336, 27922432) 27922432, 4096 00020000:00200000:1.0:1789760792.276394:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.276418:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6816, 6816] 00000008:00010000:1.0:1789760792.276437:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.276443:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.276455:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27918336, 27922432) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.276471:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27918336=0x1aa0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.276537:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.276541:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27922432 00000020:00010000:1.0:1789760792.276552:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.276556:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.276560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.276564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.276568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.276577:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.276584:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.276601:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760792.276742:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 27918336<>27922432 00000100:00100000:0.0:1789760792.276866:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495956608:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760792.324217:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760792.324242:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.324254:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.324274:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760792.324303:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495956608:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.324401:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6816, 6816] 00000080:00200000:1.0:1789760792.324419:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.324426:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.324442:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27922432 i_blocks 55305 00000020:00200000:1.0:1789760792.324446:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.324456:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.324492:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27922432, count: 4096 00000080:00200000:1.0:1789760792.324499:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27922432, bytes: 4096 00000080:00200000:1.0:1789760792.324509:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.324518:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.324520:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27922432, 27926528) 0 0 00000080:00200000:1.0:1789760792.324525:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.324543:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6817, 6817] 00020000:00200000:1.0:1789760792.324627:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27922432 chunk: [27922432, 27926528) 27926528, 4096 00020000:00200000:1.0:1789760792.324632:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.324655:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6817, 6817] 00000008:00010000:1.0:1789760792.324673:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.324679:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.324691:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27922432, 27926528) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.324707:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27922432=0x1aa1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.324762:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.324765:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27926528 00000020:00010000:1.0:1789760792.324774:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.324777:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.324780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.324783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.324787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.324795:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.324803:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.324820:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760792.325000:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 27922432<>27926528 00000100:00100000:0.0:1789760792.325063:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956736:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.365487:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974134 gen 2 00000008:00010000:0.0:1789760792.365559:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760792.365567:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.365577:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.365598:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760792.365625:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956736:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.365842:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6817, 6817] 00000080:00200000:1.0:1789760792.365873:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.365880:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.365896:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27926528 i_blocks 55305 00000020:00200000:1.0:1789760792.365901:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.365911:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.365945:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27926528, count: 4096 00000080:00200000:1.0:1789760792.365952:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27926528, bytes: 4096 00000080:00200000:1.0:1789760792.365962:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.365971:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.365974:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27926528, 27930624) 0 0 00000080:00200000:1.0:1789760792.365978:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.365994:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6818, 6818] 00020000:00200000:1.0:1789760792.366003:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27926528 chunk: [27926528, 27930624) 27930624, 4096 00020000:00200000:1.0:1789760792.366008:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.366029:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6818, 6818] 00000008:00010000:1.0:1789760792.366785:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.366792:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.366805:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27926528, 27930624) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.366820:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27926528=0x1aa2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.366893:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.366896:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27930624 00000020:00010000:1.0:1789760792.366904:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.366908:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.366910:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.366913:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.366916:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.366924:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.366931:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.366952:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760792.367083:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 27926528<>27930624 00000100:00100000:0.0:1789760792.367912:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495956864:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.402675:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974135 gen 2 00000008:00010000:0.0:1789760792.402726:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760792.402734:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.402745:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.403411:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760792.403443:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495956864:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.403616:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6818, 6818] 00000080:00200000:1.0:1789760792.403634:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.403640:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.403656:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27930624 i_blocks 55305 00000020:00200000:1.0:1789760792.403660:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.403669:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.403705:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27930624, count: 4096 00000080:00200000:1.0:1789760792.403712:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27930624, bytes: 4096 00000080:00200000:1.0:1789760792.403722:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.403802:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.403805:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27930624, 27934720) 0 0 00000080:00200000:1.0:1789760792.403810:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.403828:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6819, 6819] 00020000:00200000:1.0:1789760792.403840:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27930624 chunk: [27930624, 27934720) 27934720, 4096 00020000:00200000:1.0:1789760792.403845:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.403866:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6819, 6819] 00000008:00010000:1.0:1789760792.403884:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.403890:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.403901:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27930624, 27934720) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.403915:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27930624=0x1aa3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.403970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.403973:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27934720 00000020:00010000:1.0:1789760792.403981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.403985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.403989:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.403992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.403996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.404003:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.404011:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.404617:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760792.404877:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 27930624<>27934720 00000100:00100000:0.0:1789760792.404947:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956992:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.466886:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974136 gen 2 00000008:00010000:0.0:1789760792.466937:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760792.466946:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.466957:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.467018:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760792.467047:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495956992:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.467130:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6819, 6819] 00000080:00200000:1.0:1789760792.467146:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.467152:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.467190:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27934720 i_blocks 55305 00000020:00200000:1.0:1789760792.467194:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.467204:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.467361:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27934720, count: 4096 00000080:00200000:1.0:1789760792.467369:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27934720, bytes: 4096 00000080:00200000:1.0:1789760792.467382:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.467392:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.467394:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27934720, 27938816) 0 0 00000080:00200000:1.0:1789760792.467399:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.467496:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6820, 6820] 00020000:00200000:1.0:1789760792.467506:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27934720 chunk: [27934720, 27938816) 27938816, 4096 00020000:00200000:1.0:1789760792.467511:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.467531:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6820, 6820] 00000008:00010000:1.0:1789760792.467549:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.467555:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.467567:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27934720, 27938816) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.467581:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27934720=0x1aa4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.467637:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.467639:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27938816 00000020:00010000:1.0:1789760792.467648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.467652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.467655:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.467659:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.467662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.467671:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.467679:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.467696:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760792.467824:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 27934720<>27938816 00000100:00100000:0.0:1789760792.467905:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957120:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760792.513878:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760792.513896:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.513908:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.513931:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760792.513963:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957120:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.514239:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6820, 6820] 00000080:00200000:1.0:1789760792.514255:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.514262:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.514277:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27938816 i_blocks 55305 00000020:00200000:1.0:1789760792.514281:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.514289:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.514322:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27938816, count: 4096 00000080:00200000:1.0:1789760792.514330:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27938816, bytes: 4096 00000080:00200000:1.0:1789760792.514339:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.514348:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.514351:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27938816, 27942912) 0 0 00000080:00200000:1.0:1789760792.514355:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.514373:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6821, 6821] 00020000:00200000:1.0:1789760792.514381:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27938816 chunk: [27938816, 27942912) 27942912, 4096 00020000:00200000:1.0:1789760792.514386:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.514407:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6821, 6821] 00000008:00010000:1.0:1789760792.514425:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.514431:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.514628:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27938816, 27942912) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.514644:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27938816=0x1aa5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.514702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.514705:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27942912 00000020:00010000:1.0:1789760792.514714:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.514717:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.514720:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.514724:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.514727:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.514735:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.514742:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.514762:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760792.514900:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 27938816<>27942912 00000100:00100000:0.0:1789760792.515069:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760792.515091:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957248:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.556807:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974138 gen 2 00000008:00010000:0.0:1789760792.556873:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760792.556881:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.556892:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.556916:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760792.556949:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957248:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.556952:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6821, 6821] 00000080:00200000:1.0:1789760792.556972:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.556979:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.556996:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27942912 i_blocks 55305 00000020:00200000:1.0:1789760792.557000:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.557087:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.557127:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27942912, count: 4096 00000080:00200000:1.0:1789760792.557135:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27942912, bytes: 4096 00000080:00200000:1.0:1789760792.557147:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.557156:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.557158:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27942912, 27947008) 0 0 00000080:00200000:1.0:1789760792.557163:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.557284:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6822, 6822] 00020000:00200000:1.0:1789760792.557293:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27942912 chunk: [27942912, 27947008) 27947008, 4096 00020000:00200000:1.0:1789760792.557298:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.557318:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6822, 6822] 00000008:00010000:1.0:1789760792.557335:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.557342:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.557353:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27942912, 27947008) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.557367:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27942912=0x1aa6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.557422:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.557425:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27947008 00000020:00010000:1.0:1789760792.557433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.557437:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.557440:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.557444:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.557447:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.557455:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.557462:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.557480:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760792.557608:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 27942912<>27947008 00000100:00100000:0.0:1789760792.557724:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760792.557742:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495957376:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.584223:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974139 gen 2 00000008:00010000:0.0:1789760792.584279:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760792.584287:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.584297:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.584317:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760792.584345:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495957376:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.584483:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6822, 6822] 00000080:00200000:1.0:1789760792.584501:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.584508:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.584523:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27947008 i_blocks 55305 00000020:00200000:1.0:1789760792.584527:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.584536:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.584583:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27947008, count: 4096 00000080:00200000:1.0:1789760792.584591:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27947008, bytes: 4096 00000080:00200000:1.0:1789760792.584600:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.584609:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.584612:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27947008, 27951104) 0 0 00000080:00200000:1.0:1789760792.584616:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.584633:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6823, 6823] 00020000:00200000:1.0:1789760792.584642:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27947008 chunk: [27947008, 27951104) 27951104, 4096 00020000:00200000:1.0:1789760792.584646:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.584666:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6823, 6823] 00000008:00010000:1.0:1789760792.584684:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.584690:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.584702:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27947008, 27951104) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.584716:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27947008=0x1aa7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.584774:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.584777:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27951104 00000020:00010000:1.0:1789760792.584785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.584789:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.584793:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.584797:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.584801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.584809:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.584817:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.584833:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760792.584951:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 27947008<>27951104 00000100:00100000:0.0:1789760792.585350:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495957504:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760792.624409:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760792.624428:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.624440:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.624497:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760792.624524:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495957504:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.624682:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6823, 6823] 00000080:00200000:1.0:1789760792.624709:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.624716:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.624731:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27951104 i_blocks 55305 00000020:00200000:1.0:1789760792.624735:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.624744:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.624776:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27951104, count: 4096 00000080:00200000:1.0:1789760792.624783:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27951104, bytes: 4096 00000080:00200000:1.0:1789760792.624793:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.624801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.624804:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27951104, 27955200) 0 0 00000080:00200000:1.0:1789760792.624808:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.624823:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6824, 6824] 00020000:00200000:1.0:1789760792.624831:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27951104 chunk: [27951104, 27955200) 27955200, 4096 00020000:00200000:1.0:1789760792.624835:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.624852:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6824, 6824] 00000008:00010000:1.0:1789760792.624867:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.624871:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.624882:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27951104, 27955200) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.624895:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27951104=0x1aa8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.624942:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.624952:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27955200 00000020:00010000:1.0:1789760792.624960:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.624963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.624966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.624969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.624972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.624979:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.625382:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.625403:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760792.625525:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 27951104<>27955200 00000100:00100000:0.0:1789760792.625688:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957632:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.658493:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974141 gen 2 00000008:00010000:0.0:1789760792.658615:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760792.658624:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.658635:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.658654:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760792.658683:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957632:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.658784:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6824, 6824] 00000080:00200000:1.0:1789760792.658803:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.658810:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.658826:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27955200 i_blocks 55305 00000020:00200000:1.0:1789760792.658830:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.658840:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.658874:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27955200, count: 4096 00000080:00200000:1.0:1789760792.658884:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27955200, bytes: 4096 00000080:00200000:1.0:1789760792.658893:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.658902:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.658905:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27955200, 27959296) 0 0 00000080:00200000:1.0:1789760792.658909:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.658926:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6825, 6825] 00020000:00200000:1.0:1789760792.658934:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27955200 chunk: [27955200, 27959296) 27959296, 4096 00020000:00200000:1.0:1789760792.658939:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.658958:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6825, 6825] 00000008:00010000:1.0:1789760792.658977:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.658983:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.658993:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27955200, 27959296) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.659008:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27955200=0x1aa9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.659057:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.659060:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27959296 00000020:00010000:1.0:1789760792.659068:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.659071:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.659074:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.659077:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.659081:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.659089:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.659096:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.659333:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:0.0:1789760792.659456:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27955200<>27959296 00000100:00100000:0.0:1789760792.659488:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760792.659503:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957760:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760792.701928:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760792.701943:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.701955:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.702020:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760792.702059:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957760:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760792.702468:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6825, 6825] 00000080:00200000:1.0:1789760792.702485:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.702492:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.702507:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27959296 i_blocks 55305 00000020:00200000:1.0:1789760792.702511:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.702521:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.702553:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27959296, count: 4096 00000080:00200000:1.0:1789760792.702560:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27959296, bytes: 4096 00000080:00200000:1.0:1789760792.702569:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.702577:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.702580:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27959296, 27963392) 0 0 00000080:00200000:1.0:1789760792.702584:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.702599:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6826, 6826] 00020000:00200000:1.0:1789760792.702607:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27959296 chunk: [27959296, 27963392) 27963392, 4096 00020000:00200000:1.0:1789760792.702612:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.702630:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6826, 6826] 00000008:00010000:1.0:1789760792.702647:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.702652:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760792.702664:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27959296, 27963392) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.702677:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27959296=0x1aaa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.702733:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.702736:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27963392 00000020:00010000:1.0:1789760792.702744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.702748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.702751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.702755:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.702757:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.702764:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.702772:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.702789:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760792.702934:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 27959296<>27963392 00000100:00100000:0.0:1789760792.703007:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957888:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.734664:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974143 gen 2 00000008:00010000:0.0:1789760792.734718:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760792.734724:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.734732:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.734748:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760792.734781:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495957888:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760792.734795:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6826, 6826] 00000080:00200000:2.0:1789760792.734814:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.734820:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.734836:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27963392 i_blocks 55305 00000020:00200000:2.0:1789760792.734840:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.734912:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.734948:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27963392, count: 4096 00000080:00200000:2.0:1789760792.734955:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27963392, bytes: 4096 00000080:00200000:2.0:1789760792.734965:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.734974:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.734977:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27963392, 27967488) 0 0 00000080:00200000:2.0:1789760792.734981:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.734996:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6827, 6827] 00020000:00200000:2.0:1789760792.735004:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27963392 chunk: [27963392, 27967488) 27967488, 4096 00020000:00200000:2.0:1789760792.735009:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.735030:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6827, 6827] 00000008:00010000:2.0:1789760792.735167:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.735206:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760792.735218:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27963392, 27967488) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.735245:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27963392=0x1aab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.735299:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.735302:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27967488 00000020:00010000:2.0:1789760792.735310:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.735314:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.735316:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.735320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.735323:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.735330:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.735337:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.735354:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760792.735476:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 27963392<>27967488 00000100:00100000:2.0:1789760792.735526:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760792.735541:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495958016:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760792.769775:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974144 gen 2 00000008:00010000:2.0:1789760792.769826:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:2.0:1789760792.769835:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.769846:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.769927:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:2.0:1789760792.769958:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495958016:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760792.770152:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6827, 6827] 00000080:00200000:3.0:1789760792.770324:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.770331:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.770425:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27967488 i_blocks 55305 00000020:00200000:2.0:1789760792.770430:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.770440:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.770476:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27967488, count: 4096 00000080:00200000:2.0:1789760792.770483:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27967488, bytes: 4096 00000080:00200000:2.0:1789760792.770493:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.770501:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.770504:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27967488, 27971584) 0 0 00000080:00200000:2.0:1789760792.770508:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.770524:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6828, 6828] 00020000:00200000:2.0:1789760792.770533:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27967488 chunk: [27967488, 27971584) 27971584, 4096 00020000:00200000:2.0:1789760792.770538:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.770559:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6828, 6828] 00000008:00010000:2.0:1789760792.770578:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.770584:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760792.770597:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27967488, 27971584) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.770612:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27967488=0x1aac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.770667:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.770670:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27971584 00000020:00010000:2.0:1789760792.770678:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.770682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.770685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.770688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.770691:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.770699:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.770706:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.770754:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760792.770944:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 27967488<>27971584 00000100:00100000:3.0:1789760792.771015:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495958144:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760792.799090:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760792.799107:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.799118:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.799143:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000080:00200000:0.0:1789760792.799347:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6828, 6828] 00000080:00200000:0.0:1789760792.799367:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.799374:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.799390:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27971584 i_blocks 55305 00000020:00200000:0.0:1789760792.799393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.799403:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.799439:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27971584, count: 4096 00000080:00200000:0.0:1789760792.799446:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27971584, bytes: 4096 00000080:00200000:0.0:1789760792.799456:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.799464:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.799466:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27971584, 27975680) 0 0 00000080:00200000:0.0:1789760792.799471:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.799487:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6829, 6829] 00020000:00200000:0.0:1789760792.799495:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27971584 chunk: [27971584, 27975680) 27975680, 4096 00020000:00200000:0.0:1789760792.799499:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.799517:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6829, 6829] 00000008:00010000:0.0:1789760792.799533:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.799538:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760792.799558:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27971584, 27975680) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.799574:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27971584=0x1aad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.799625:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.799627:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27975680 00000020:00010000:0.0:1789760792.799636:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.799640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.799643:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.799646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.799648:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.799655:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.799662:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760792.799713:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495958144:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760792.799748:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ec00 00000008:00100000:3.0:1789760792.799885:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145f80 - object 0x240000400:4 offset 27971584<>27975680 00000100:00100000:2.0:1789760792.799968:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495958272:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760792.844538:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974146 gen 2 00000008:00010000:2.0:1789760792.844599:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ec00 00000020:00200000:2.0:1789760792.844607:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.844618:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.844642:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ec00 00000100:00100000:2.0:1789760792.844672:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495958272:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760792.844688:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6829, 6829] 00000080:00200000:0.0:1789760792.844707:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.844714:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.844730:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27975680 i_blocks 55305 00000020:00200000:0.0:1789760792.844735:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.844746:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.844779:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27975680, count: 4096 00000080:00200000:0.0:1789760792.844787:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27975680, bytes: 4096 00000080:00200000:0.0:1789760792.844798:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.844807:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.844810:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27975680, 27979776) 0 0 00000080:00200000:0.0:1789760792.844814:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.844830:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6830, 6830] 00020000:00200000:0.0:1789760792.844838:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27975680 chunk: [27975680, 27979776) 27979776, 4096 00020000:00200000:0.0:1789760792.844843:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.844863:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6830, 6830] 00000008:00010000:0.0:1789760792.844881:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.844887:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760792.844898:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27975680, 27979776) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.844912:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27975680=0x1aae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.844970:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.844973:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27979776 00000020:00010000:0.0:1789760792.844982:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.844985:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.844988:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.844992:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.844995:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.845002:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.845009:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.845026:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760792.845149:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 27975680<>27979776 00000100:00100000:1.0:1789760792.845318:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495958400:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760792.883144:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974147 gen 2 00000008:00010000:0.0:1789760792.883247:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760792.883257:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.883270:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.883299:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760792.883329:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495958400:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760792.883603:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6830, 6830] 00000080:00200000:2.0:1789760792.883620:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.883626:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.883643:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27979776 i_blocks 55305 00000020:00200000:2.0:1789760792.883647:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.883656:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.883689:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27979776, count: 4096 00000080:00200000:2.0:1789760792.883696:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27979776, bytes: 4096 00000080:00200000:2.0:1789760792.883711:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.883719:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.883721:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27979776, 27983872) 0 0 00000080:00200000:2.0:1789760792.883725:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.883743:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6831, 6831] 00020000:00200000:2.0:1789760792.883752:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27979776 chunk: [27979776, 27983872) 27983872, 4096 00020000:00200000:2.0:1789760792.883757:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.883788:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6831, 6831] 00000008:00010000:2.0:1789760792.883809:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.883815:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760792.883827:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27979776, 27983872) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.883842:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27979776=0x1aaf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.883899:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.883902:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27983872 00000020:00010000:2.0:1789760792.883910:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.884604:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.884610:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.884614:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088143f8 (ffff8dd205fe0878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.884617:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.884629:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.884638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.884664:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760792.884807:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 27979776<>27983872 00000100:00100000:2.0:1789760792.884860:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760792.884883:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495958528:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760792.950345:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974148 gen 2 00000008:00010000:2.0:1789760792.950404:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760792.950413:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.950424:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.950454:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760792.950486:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495958528:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760792.950621:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6831, 6831] 00000080:00200000:3.0:1789760792.950638:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.950645:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.950662:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27983872 i_blocks 55305 00000020:00200000:3.0:1789760792.950666:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.950678:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.950750:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27983872, count: 4096 00000080:00200000:3.0:1789760792.950758:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27983872, bytes: 4096 00000080:00200000:3.0:1789760792.950773:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.950781:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760792.950784:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27983872, 27987968) 0 0 00000080:00200000:3.0:1789760792.950789:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.950805:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6832, 6832] 00020000:00200000:3.0:1789760792.950813:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27983872 chunk: [27983872, 27987968) 27987968, 4096 00020000:00200000:3.0:1789760792.950819:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.950838:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6832, 6832] 00000008:00010000:3.0:1789760792.950858:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.950864:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760792.950875:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27983872, 27987968) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.950891:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27983872=0x1ab0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.950952:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.950954:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27987968 00000020:00010000:3.0:1789760792.950963:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.950966:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.950969:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.950973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.950975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.950982:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.950988:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.951005:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760792.951127:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 27983872<>27987968 00000100:00100000:3.0:1789760792.951217:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.951246:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495958656:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760792.990147:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974149 gen 2 00000008:00010000:3.0:1789760792.990246:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:3.0:1789760792.990254:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.990265:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.990348:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:3.0:1789760792.990382:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495958656:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760792.990495:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6832, 6832] 00000080:00200000:2.0:1789760792.990512:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.990519:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.990534:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27987968 i_blocks 55305 00000020:00200000:2.0:1789760792.990538:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.990547:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.990582:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27987968, count: 4096 00000080:00200000:2.0:1789760792.990589:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27987968, bytes: 4096 00000080:00200000:2.0:1789760792.990599:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.990608:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.990610:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27987968, 27992064) 0 0 00000080:00200000:2.0:1789760792.990614:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.990631:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6833, 6833] 00020000:00200000:2.0:1789760792.990639:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27987968 chunk: [27987968, 27992064) 27992064, 4096 00020000:00200000:2.0:1789760792.990643:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.990661:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6833, 6833] 00000008:00010000:2.0:1789760792.991347:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.991354:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760792.991367:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27987968, 27992064) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.991383:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27987968=0x1ab1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.991446:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.991449:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27992064 00000020:00010000:2.0:1789760792.991458:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.991462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.991465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.991469:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.991472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.991479:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.991487:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.991510:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760792.991698:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 27987968<>27992064 00000100:00100000:2.0:1789760792.991813:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760792.991834:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495958784:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760793.033655:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760793.033672:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.033685:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.033825:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760793.033860:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495958784:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760793.034384:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6833, 6833] 00000080:00200000:3.0:1789760793.034402:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.034409:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.034423:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27992064 i_blocks 55305 00000020:00200000:3.0:1789760793.034429:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.034439:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.034471:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27992064, count: 4096 00000080:00200000:3.0:1789760793.034478:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27992064, bytes: 4096 00000080:00200000:3.0:1789760793.034489:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.034498:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.034501:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27992064, 27996160) 0 0 00000080:00200000:3.0:1789760793.034506:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.034522:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6834, 6834] 00020000:00200000:3.0:1789760793.034530:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27992064 chunk: [27992064, 27996160) 27996160, 4096 00020000:00200000:3.0:1789760793.034535:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.034555:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6834, 6834] 00000008:00010000:3.0:1789760793.034572:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.034578:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760793.034590:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27992064, 27996160) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.034606:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27992064=0x1ab2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.034662:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.034665:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27996160 00000020:00010000:3.0:1789760793.034673:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.034677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.034680:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.034683:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.034686:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.034694:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.034701:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.034719:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760793.035988:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144a80 - object 0x240000400:4 offset 27992064<>27996160 00000100:00100000:3.0:1789760793.036117:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.036141:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495958912:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760793.036448:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760793.064578:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974150 gen 2 00000100:00100000:3.0:1789760793.064618:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203144a80 x1876699495958912/t4294974151(4294974151) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.064666:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:3.0:1789760793.064676:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.064686:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.064710:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:3.0:1789760793.064736:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495958912:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760793.064883:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6834, 6834] 00000080:00200000:2.0:1789760793.064900:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.064906:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.064921:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27996160 i_blocks 55305 00000020:00200000:2.0:1789760793.064924:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.064933:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.064968:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27996160, count: 4096 00000080:00200000:2.0:1789760793.064975:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27996160, bytes: 4096 00000080:00200000:2.0:1789760793.064984:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.064993:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.064995:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27996160, 28000256) 0 0 00000080:00200000:2.0:1789760793.065000:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.065015:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6835, 6835] 00020000:00200000:2.0:1789760793.065024:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27996160 chunk: [27996160, 28000256) 28000256, 4096 00020000:00200000:2.0:1789760793.065028:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.065047:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6835, 6835] 00000008:00010000:2.0:1789760793.065065:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.065071:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.065082:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27996160, 28000256) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.065095:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=27996160=0x1ab3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.065148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.065151:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28000256 00000020:00010000:2.0:1789760793.065159:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.065163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.065166:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.065214:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.065217:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.065233:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.065240:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.065254:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760793.065368:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 27996160<>28000256 00000100:00100000:2.0:1789760793.065408:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760793.065424:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495959040:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760793.106437:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974152 gen 2 00000008:00010000:2.0:1789760793.106500:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760793.106509:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.106519:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.107363:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760793.107403:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495959040:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760793.107548:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6835, 6835] 00000080:00200000:3.0:1789760793.107565:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.107571:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.107586:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28000256 i_blocks 55305 00000020:00200000:3.0:1789760793.107590:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.107608:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.107642:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28000256, count: 4096 00000080:00200000:3.0:1789760793.107649:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28000256, bytes: 4096 00000080:00200000:3.0:1789760793.107662:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.107672:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.107674:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28000256, 28004352) 0 0 00000080:00200000:3.0:1789760793.107679:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.107695:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6836, 6836] 00020000:00200000:3.0:1789760793.107704:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28000256 chunk: [28000256, 28004352) 28004352, 4096 00020000:00200000:3.0:1789760793.107708:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.107727:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6836, 6836] 00000008:00010000:3.0:1789760793.107750:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.107756:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760793.107767:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28000256, 28004352) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.107780:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28000256=0x1ab4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.107835:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.107838:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28004352 00000020:00010000:3.0:1789760793.107846:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.107850:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.107853:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.107857:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.107860:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.107867:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.107883:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.107901:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760793.108024:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 28000256<>28004352 00000100:00100000:3.0:1789760793.108067:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.108083:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495959168:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760793.152580:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974153 gen 2 00000008:00010000:3.0:1789760793.152668:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:3.0:1789760793.152685:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.152707:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.152751:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:3.0:1789760793.152795:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495959168:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760793.153015:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6836, 6836] 00000080:00200000:2.0:1789760793.153038:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.153044:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.153064:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28004352 i_blocks 55305 00000020:00200000:2.0:1789760793.153072:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.153089:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.153145:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28004352, count: 4096 00000080:00200000:2.0:1789760793.153159:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28004352, bytes: 4096 00000080:00200000:2.0:1789760793.153238:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.153253:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.153256:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28004352, 28008448) 0 0 00000080:00200000:2.0:1789760793.153263:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.153289:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6837, 6837] 00020000:00200000:2.0:1789760793.153303:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28004352 chunk: [28004352, 28008448) 28008448, 4096 00020000:00200000:2.0:1789760793.153372:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.153394:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6837, 6837] 00000008:00010000:2.0:1789760793.153413:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.153419:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.153431:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28004352, 28008448) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.153456:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28004352=0x1ab5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.153568:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.153573:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28008448 00000020:00010000:2.0:1789760793.153586:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.153592:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.153595:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.153602:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.153607:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.153618:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.153628:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.153653:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760793.153816:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 28004352<>28008448 00000100:00100000:3.0:1789760793.153894:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495959296:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760793.184093:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974154 gen 2 00000008:00010000:3.0:1789760793.184142:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760793.184150:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.184160:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.184210:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760793.184247:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495959296:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760793.184416:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6837, 6837] 00000080:00200000:2.0:1789760793.184433:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.184440:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.184455:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28008448 i_blocks 55305 00000020:00200000:2.0:1789760793.184459:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.184469:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.184500:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28008448, count: 4096 00000080:00200000:2.0:1789760793.184508:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28008448, bytes: 4096 00000080:00200000:2.0:1789760793.184519:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.184528:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.184531:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28008448, 28012544) 0 0 00000080:00200000:2.0:1789760793.184535:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.184621:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6838, 6838] 00020000:00200000:2.0:1789760793.184630:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28008448 chunk: [28008448, 28012544) 28012544, 4096 00020000:00200000:2.0:1789760793.184635:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.184654:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6838, 6838] 00000008:00010000:2.0:1789760793.184672:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.184677:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.184689:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28008448, 28012544) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.184702:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28008448=0x1ab6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.184758:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.184761:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28012544 00000020:00010000:2.0:1789760793.184769:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.184772:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.184775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.184778:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.184781:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.184788:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.184795:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.184810:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760793.184942:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 28008448<>28012544 00000100:00100000:3.0:1789760793.185013:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495959424:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760793.218788:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974155 gen 2 00000008:00010000:3.0:1789760793.218837:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760793.218846:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.218859:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.218880:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760793.218908:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495959424:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760793.219059:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6838, 6838] 00000080:00200000:1.0:1789760793.219075:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.219082:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.219099:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28012544 i_blocks 55305 00000020:00200000:1.0:1789760793.219103:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.219113:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.219145:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28012544, count: 4096 00000080:00200000:1.0:1789760793.219152:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28012544, bytes: 4096 00000080:00200000:1.0:1789760793.219162:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.219212:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760793.219214:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28012544, 28016640) 0 0 00000080:00200000:1.0:1789760793.219218:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.219244:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6839, 6839] 00020000:00200000:1.0:1789760793.219253:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28012544 chunk: [28012544, 28016640) 28016640, 4096 00020000:00200000:1.0:1789760793.219258:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.219278:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6839, 6839] 00000008:00010000:1.0:1789760793.219296:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.219301:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760793.219313:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28012544, 28016640) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.219327:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28012544=0x1ab7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.219384:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.219387:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28016640 00000020:00010000:1.0:1789760793.219396:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.219399:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.219402:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.219406:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.219409:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.219416:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.219424:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.219439:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760793.219668:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 28012544<>28016640 00000100:00100000:0.0:1789760793.219744:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495959552:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760793.251665:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760793.251682:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.251694:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.251714:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760793.251741:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495959552:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760793.251831:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6839, 6839] 00000080:00200000:1.0:1789760793.251899:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.251906:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.251923:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28016640 i_blocks 55305 00000020:00200000:1.0:1789760793.251926:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.251936:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.251969:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28016640, count: 4096 00000080:00200000:1.0:1789760793.251975:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28016640, bytes: 4096 00000080:00200000:1.0:1789760793.251985:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.251994:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760793.251997:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28016640, 28020736) 0 0 00000080:00200000:1.0:1789760793.252001:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.252017:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6840, 6840] 00020000:00200000:1.0:1789760793.252025:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28016640 chunk: [28016640, 28020736) 28020736, 4096 00020000:00200000:1.0:1789760793.252030:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.252048:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6840, 6840] 00000008:00010000:1.0:1789760793.252066:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.252072:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760793.252084:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28016640, 28020736) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.252098:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28016640=0x1ab8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.252152:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.252155:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28020736 00000020:00010000:1.0:1789760793.252163:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.252307:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.252313:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.252317:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.252320:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.252329:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.252337:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.252353:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760793.252476:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 28016640<>28020736 00000100:00100000:0.0:1789760793.252538:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495959680:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760793.286839:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974157 gen 2 00000008:00010000:0.0:1789760793.286905:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760793.286914:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.286924:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.286946:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760793.286979:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495959680:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760793.287082:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6840, 6840] 00000080:00200000:1.0:1789760793.287101:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.287107:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.287122:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28020736 i_blocks 55305 00000020:00200000:1.0:1789760793.287126:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.287135:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.287204:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28020736, count: 4096 00000080:00200000:1.0:1789760793.287211:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28020736, bytes: 4096 00000080:00200000:1.0:1789760793.287233:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.287250:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760793.287252:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28020736, 28024832) 0 0 00000080:00200000:1.0:1789760793.287258:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.287276:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6841, 6841] 00020000:00200000:1.0:1789760793.287283:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28020736 chunk: [28020736, 28024832) 28024832, 4096 00020000:00200000:1.0:1789760793.287288:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.287307:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6841, 6841] 00000008:00010000:1.0:1789760793.287328:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.287334:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760793.287345:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28020736, 28024832) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.287360:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28020736=0x1ab9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.287422:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.287425:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28024832 00000020:00010000:1.0:1789760793.287433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.287437:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.287440:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.287444:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.287447:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.287454:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.287460:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.287476:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760793.287601:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 28020736<>28024832 00000100:00100000:0.0:1789760793.287784:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495959808:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760793.313368:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974158 gen 2 00000008:00010000:0.0:1789760793.313420:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760793.313428:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.313438:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.313497:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760793.313525:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495959808:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760793.313671:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6841, 6841] 00000080:00200000:1.0:1789760793.313687:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.313695:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.313708:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28024832 i_blocks 55305 00000020:00200000:1.0:1789760793.313711:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.313719:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.313752:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28024832, count: 4096 00000080:00200000:1.0:1789760793.313759:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28024832, bytes: 4096 00000080:00200000:1.0:1789760793.313768:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.313777:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760793.313780:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28024832, 28028928) 0 0 00000080:00200000:1.0:1789760793.313784:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.313801:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6842, 6842] 00020000:00200000:1.0:1789760793.313809:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28024832 chunk: [28024832, 28028928) 28028928, 4096 00020000:00200000:1.0:1789760793.313814:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.313834:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6842, 6842] 00000008:00010000:1.0:1789760793.313914:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.313920:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760793.313932:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28024832, 28028928) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.313946:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28024832=0x1aba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.313999:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.314002:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28028928 00000020:00010000:1.0:1789760793.314011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.314014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.314018:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.314021:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.314025:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.314033:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.314040:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.314056:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760793.314303:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e300 - object 0x240000400:4 offset 28024832<>28028928 00000100:00100000:0.0:1789760793.314357:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495959936:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760793.342777:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974159 gen 2 00000008:00010000:0.0:1789760793.342829:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760793.342837:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.342860:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.342881:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760793.342911:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495959936:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760793.343085:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6842, 6842] 00000080:00200000:3.0:1789760793.343103:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.343109:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.343127:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28028928 i_blocks 55305 00000020:00200000:3.0:1789760793.343131:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.343141:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.343218:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28028928, count: 4096 00000080:00200000:3.0:1789760793.343236:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28028928, bytes: 4096 00000080:00200000:3.0:1789760793.343247:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.343256:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.343258:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28028928, 28033024) 0 0 00000080:00200000:3.0:1789760793.343263:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.343279:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6843, 6843] 00020000:00200000:3.0:1789760793.343287:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28028928 chunk: [28028928, 28033024) 28033024, 4096 00020000:00200000:3.0:1789760793.343292:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.343313:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6843, 6843] 00000008:00010000:3.0:1789760793.343330:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.343336:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760793.343347:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28028928, 28033024) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.343361:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28028928=0x1abb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.343415:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.343418:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28033024 00000020:00010000:3.0:1789760793.343426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.343430:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.343433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.343436:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.343439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.343446:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.343453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.343467:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760793.343588:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145180 - object 0x240000400:4 offset 28028928<>28033024 00000100:00100000:2.0:1789760793.343661:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495960064:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760793.375522:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974160 gen 2 00000008:00010000:2.0:1789760793.375574:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760793.375589:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.375599:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.375621:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000080:00200000:3.0:1789760793.375684:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6843, 6843] 00000080:00200000:3.0:1789760793.375703:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.375710:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.375727:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28033024 i_blocks 55305 00000020:00200000:3.0:1789760793.375731:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.375741:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.375775:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28033024, count: 4096 00000080:00200000:3.0:1789760793.375783:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28033024, bytes: 4096 00000080:00200000:3.0:1789760793.375793:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.375802:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.375804:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28033024, 28037120) 0 0 00000080:00200000:3.0:1789760793.375808:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.375824:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6844, 6844] 00020000:00200000:3.0:1789760793.375832:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28033024 chunk: [28033024, 28037120) 28037120, 4096 00020000:00200000:3.0:1789760793.375837:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.375870:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6844, 6844] 00000008:00010000:3.0:1789760793.375886:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.375892:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760793.375904:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28033024, 28037120) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.375918:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28033024=0x1abc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.376108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.376112:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28037120 00000020:00010000:3.0:1789760793.376120:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.376124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.376127:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.376130:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.376134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.376142:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.376148:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1789760793.377805:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495960064:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760793.377878:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:2.0:1789760793.378021:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 28033024<>28037120 00000100:00100000:2.0:1789760793.378074:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760793.378090:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495960192:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760793.436584:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974161 gen 2 00000008:00010000:2.0:1789760793.436635:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760793.436643:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.436656:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.436715:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760793.436755:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495960192:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760793.439965:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6844, 6844] 00000080:00200000:3.0:1789760793.439994:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.440001:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.440019:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28037120 i_blocks 55305 00000020:00200000:3.0:1789760793.440024:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.440040:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.440083:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28037120, count: 4096 00000080:00200000:3.0:1789760793.440093:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28037120, bytes: 4096 00000080:00200000:3.0:1789760793.440110:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.440120:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.440123:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28037120, 28041216) 0 0 00000080:00200000:3.0:1789760793.440128:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.440146:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6845, 6845] 00020000:00200000:3.0:1789760793.440156:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28037120 chunk: [28037120, 28041216) 28041216, 4096 00020000:00200000:3.0:1789760793.440161:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.440206:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6845, 6845] 00000008:00010000:3.0:1789760793.440282:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.440288:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760793.440300:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28037120, 28041216) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.440315:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28037120=0x1abd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.440449:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.440452:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28041216 00000020:00010000:3.0:1789760793.440566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.440571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.440574:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.440578:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.440581:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.440590:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.440599:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.440627:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760793.440831:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144380 - object 0x240000400:4 offset 28037120<>28041216 00000100:00100000:2.0:1789760793.440921:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495960320:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760793.478771:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:2.0:1789760793.478787:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.478798:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.478820:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:2.0:1789760793.478859:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495960320:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760793.479010:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6845, 6845] 00000080:00200000:3.0:1789760793.479027:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.479034:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.479050:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28041216 i_blocks 55305 00000020:00200000:3.0:1789760793.479054:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.479063:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.479100:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28041216, count: 4096 00000080:00200000:3.0:1789760793.479108:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28041216, bytes: 4096 00000080:00200000:3.0:1789760793.479118:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.479127:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.479129:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28041216, 28045312) 0 0 00000080:00200000:3.0:1789760793.479134:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.479152:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6846, 6846] 00020000:00200000:3.0:1789760793.479160:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28041216 chunk: [28041216, 28045312) 28045312, 4096 00020000:00200000:3.0:1789760793.479165:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.479216:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6846, 6846] 00000008:00010000:3.0:1789760793.479243:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.479249:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760793.479260:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28041216, 28045312) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.479275:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28041216=0x1abe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.479334:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.479337:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28045312 00000020:00010000:3.0:1789760793.479346:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.479349:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.479352:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.479356:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.479359:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.479367:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.479373:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.479391:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250000 00000008:00100000:3.0:1789760793.479513:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146300 - object 0x240000400:4 offset 28041216<>28045312 00000100:00100000:2.0:1789760793.479859:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495960448:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760793.520601:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974163 gen 2 00000008:00010000:2.0:1789760793.522859:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250000 00000020:00200000:2.0:1789760793.522873:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.522884:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.522912:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250000 00000100:00100000:2.0:1789760793.522946:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495960448:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760793.522959:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6846, 6846] 00000080:00200000:3.0:1789760793.522978:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.522984:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.522999:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28045312 i_blocks 55305 00000020:00200000:3.0:1789760793.523003:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.523012:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.523047:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28045312, count: 4096 00000080:00200000:3.0:1789760793.523054:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28045312, bytes: 4096 00000080:00200000:3.0:1789760793.523068:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.523076:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.523078:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28045312, 28049408) 0 0 00000080:00200000:3.0:1789760793.523082:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.523101:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6847, 6847] 00020000:00200000:3.0:1789760793.523109:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28045312 chunk: [28045312, 28049408) 28049408, 4096 00020000:00200000:3.0:1789760793.523114:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.523135:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6847, 6847] 00000008:00010000:3.0:1789760793.523153:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.523158:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760793.523380:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28045312, 28049408) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.523395:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28045312=0x1abf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.523450:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.523453:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28049408 00000020:00010000:3.0:1789760793.523462:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.523466:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.523469:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.523472:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.523475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.523483:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.523490:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.523506:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760793.524882:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 28045312<>28049408 00000100:00100000:3.0:1789760793.524948:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.524972:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495960576:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760793.575104:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974164 gen 2 00000008:00010000:3.0:1789760793.575153:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:3.0:1789760793.575207:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.575227:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.575247:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:3.0:1789760793.575275:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495960576:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760793.575436:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6847, 6847] 00000080:00200000:2.0:1789760793.575452:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.575458:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.575474:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28049408 i_blocks 55305 00000020:00200000:2.0:1789760793.575477:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.575488:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.575522:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28049408, count: 4096 00000080:00200000:2.0:1789760793.575529:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28049408, bytes: 4096 00000080:00200000:2.0:1789760793.575539:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.575548:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.575551:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28049408, 28053504) 0 0 00000080:00200000:2.0:1789760793.575556:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.575573:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6848, 6848] 00020000:00200000:2.0:1789760793.575582:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28049408 chunk: [28049408, 28053504) 28053504, 4096 00020000:00200000:2.0:1789760793.575587:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.575605:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6848, 6848] 00000008:00010000:2.0:1789760793.575628:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.575634:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.575647:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28049408, 28053504) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.575662:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28049408=0x1ac0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.575725:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.575728:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28053504 00000020:00010000:2.0:1789760793.575738:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.575741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.575744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.575748:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.575751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.575758:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.575765:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.575858:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760793.575980:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 28049408<>28053504 00000100:00100000:2.0:1789760793.576024:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760793.576043:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495960704:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760793.625032:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974165 gen 2 00000008:00010000:2.0:1789760793.625081:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:2.0:1789760793.625090:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.625100:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.625127:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:2.0:1789760793.625156:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495960704:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760793.625272:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6848, 6848] 00000080:00200000:0.0:1789760793.625289:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.625293:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.625307:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28053504 i_blocks 55305 00000020:00200000:0.0:1789760793.625309:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.625318:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.625355:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28053504, count: 4096 00000080:00200000:0.0:1789760793.625362:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28053504, bytes: 4096 00000080:00200000:0.0:1789760793.625369:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.625376:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.625378:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28053504, 28057600) 0 0 00000080:00200000:0.0:1789760793.625381:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.625394:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6849, 6849] 00020000:00200000:0.0:1789760793.625401:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28053504 chunk: [28053504, 28057600) 28057600, 4096 00020000:00200000:0.0:1789760793.625405:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.625421:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6849, 6849] 00000008:00010000:0.0:1789760793.625434:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.625440:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760793.625449:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28053504, 28057600) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.625462:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28053504=0x1ac1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.625517:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.625519:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28057600 00000020:00010000:0.0:1789760793.625526:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.625528:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.625531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.625872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.625877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.625886:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.625892:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.625911:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760793.626113:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 28053504<>28057600 00000100:00100000:0.0:1789760793.626162:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760793.626208:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495960832:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760793.654165:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974166 gen 2 00000008:00010000:0.0:1789760793.654479:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760793.654489:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.654500:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.654523:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760793.654559:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495960832:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760793.654690:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6849, 6849] 00000080:00200000:2.0:1789760793.654708:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.654714:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.654732:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28057600 i_blocks 55305 00000020:00200000:2.0:1789760793.654749:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.654758:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.654792:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28057600, count: 4096 00000080:00200000:2.0:1789760793.654799:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28057600, bytes: 4096 00000080:00200000:2.0:1789760793.654810:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.654819:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.654822:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28057600, 28061696) 0 0 00000080:00200000:2.0:1789760793.654826:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.654843:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6850, 6850] 00020000:00200000:2.0:1789760793.654851:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28057600 chunk: [28057600, 28061696) 28061696, 4096 00020000:00200000:2.0:1789760793.654855:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.654874:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6850, 6850] 00000008:00010000:2.0:1789760793.654892:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.654898:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.654909:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28057600, 28061696) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.654923:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28057600=0x1ac2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.654979:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.654982:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28061696 00000020:00010000:2.0:1789760793.654990:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.654993:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.654996:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.654999:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.655002:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.655009:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.655015:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.655030:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813900 00000008:00100000:2.0:1789760793.655146:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 28057600<>28061696 00000100:00100000:3.0:1789760793.655307:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.655328:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495960960:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760793.703713:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974167 gen 2 00000008:00010000:3.0:1789760793.703769:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813900 00000020:00200000:3.0:1789760793.703777:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.704397:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.704426:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813900 00000100:00100000:3.0:1789760793.704467:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495960960:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760793.705695:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6850, 6850] 00000080:00200000:2.0:1789760793.705720:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.705727:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.705745:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28061696 i_blocks 55305 00000020:00200000:2.0:1789760793.705750:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.705763:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.705799:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28061696, count: 4096 00000080:00200000:2.0:1789760793.705808:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28061696, bytes: 4096 00000080:00200000:2.0:1789760793.705825:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.705834:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.705837:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28061696, 28065792) 0 0 00000080:00200000:2.0:1789760793.705842:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.705859:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6851, 6851] 00020000:00200000:2.0:1789760793.705867:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28061696 chunk: [28061696, 28065792) 28065792, 4096 00020000:00200000:2.0:1789760793.705872:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.705894:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6851, 6851] 00000008:00010000:2.0:1789760793.705915:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.705921:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.705933:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28061696, 28065792) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.705949:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28061696=0x1ac3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.706016:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.706019:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28065792 00000020:00010000:2.0:1789760793.706028:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.706032:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.706035:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.706038:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.706041:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.706050:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.706056:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.706076:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760793.706267:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 28061696<>28065792 00000100:00100000:2.0:1789760793.706325:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495961088:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760793.753625:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974168 gen 2 00000008:00010000:2.0:1789760793.753684:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:2.0:1789760793.753821:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.753835:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.753912:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:2.0:1789760793.753942:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495961088:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760793.757376:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6851, 6851] 00000080:00200000:3.0:1789760793.757406:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.757414:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.757433:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28065792 i_blocks 55305 00000020:00200000:3.0:1789760793.757438:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.757457:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.757494:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28065792, count: 4096 00000080:00200000:3.0:1789760793.757502:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28065792, bytes: 4096 00000080:00200000:3.0:1789760793.757518:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.757528:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.757530:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28065792, 28069888) 0 0 00000080:00200000:3.0:1789760793.757535:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.757552:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6852, 6852] 00020000:00200000:3.0:1789760793.757561:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28065792 chunk: [28065792, 28069888) 28069888, 4096 00020000:00200000:3.0:1789760793.757565:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.757586:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6852, 6852] 00000008:00010000:3.0:1789760793.757604:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.757611:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760793.757622:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28065792, 28069888) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.757637:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28065792=0x1ac4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.758444:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.758451:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28069888 00000020:00010000:3.0:1789760793.758464:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.758467:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.758471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.758474:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.758477:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.758487:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.758495:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.758530:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250c00 00000008:00100000:3.0:1789760793.758672:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145c00 - object 0x240000400:4 offset 28065792<>28069888 00000100:00100000:2.0:1789760793.759434:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495961216:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760793.817739:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974169 gen 2 00000008:00010000:2.0:1789760793.817789:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250c00 00000020:00200000:2.0:1789760793.817798:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.817810:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.817837:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250c00 00000100:00100000:2.0:1789760793.817930:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495961216:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760793.818453:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6852, 6852] 00000080:00200000:0.0:1789760793.818471:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.818478:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.818494:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28069888 i_blocks 55305 00000020:00200000:0.0:1789760793.818498:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.818508:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.818596:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28069888, count: 4096 00000080:00200000:0.0:1789760793.818604:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28069888, bytes: 4096 00000080:00200000:0.0:1789760793.818616:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.818624:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.818626:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28069888, 28073984) 0 0 00000080:00200000:0.0:1789760793.818631:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.818646:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6853, 6853] 00020000:00200000:0.0:1789760793.818655:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28069888 chunk: [28069888, 28073984) 28073984, 4096 00020000:00200000:0.0:1789760793.818660:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.818680:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6853, 6853] 00000008:00010000:0.0:1789760793.818698:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.818704:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760793.818716:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28069888, 28073984) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.818730:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28069888=0x1ac5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.818790:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.818793:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28073984 00000020:00010000:0.0:1789760793.818801:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.818804:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.818807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.818810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.818812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.818820:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.818828:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.818845:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760793.819035:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 28069888<>28073984 00000100:00100000:1.0:1789760793.819118:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495961344:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760793.861252:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974170 gen 2 00000008:00010000:0.0:1789760793.861649:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760793.861658:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.861670:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.861776:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760793.861810:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495961344:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760793.861908:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6853, 6853] 00000080:00200000:2.0:1789760793.861925:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.861931:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.861946:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28073984 i_blocks 55305 00000020:00200000:2.0:1789760793.861949:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.861958:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.861992:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28073984, count: 4096 00000080:00200000:2.0:1789760793.861999:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28073984, bytes: 4096 00000080:00200000:2.0:1789760793.862010:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.862018:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.862021:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28073984, 28078080) 0 0 00000080:00200000:2.0:1789760793.862025:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.862041:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6854, 6854] 00020000:00200000:2.0:1789760793.862049:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28073984 chunk: [28073984, 28078080) 28078080, 4096 00020000:00200000:2.0:1789760793.862054:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.862072:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6854, 6854] 00000008:00010000:2.0:1789760793.862088:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.862093:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.862103:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28073984, 28078080) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.862116:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28073984=0x1ac6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.862250:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.862253:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28078080 00000020:00010000:2.0:1789760793.862262:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.862265:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.862269:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.862272:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.862275:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.862339:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.862357:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.862379:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760793.862922:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 28073984<>28078080 00000100:00100000:2.0:1789760793.862974:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760793.862992:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495961472:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760793.896974:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974171 gen 2 00000008:00010000:2.0:1789760793.897595:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:2.0:1789760793.897603:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.897612:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.897686:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:2.0:1789760793.897710:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495961472:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760793.897864:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6854, 6854] 00000080:00200000:3.0:1789760793.897877:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.897882:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.898434:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28078080 i_blocks 55305 00000020:00200000:2.0:1789760793.898440:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.898451:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.898479:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28078080, count: 4096 00000080:00200000:2.0:1789760793.898487:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28078080, bytes: 4096 00000080:00200000:2.0:1789760793.898496:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.898504:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.898506:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28078080, 28082176) 0 0 00000080:00200000:2.0:1789760793.898510:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.898524:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6855, 6855] 00020000:00200000:2.0:1789760793.898532:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28078080 chunk: [28078080, 28082176) 28082176, 4096 00020000:00200000:2.0:1789760793.898535:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.898552:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6855, 6855] 00000008:00010000:2.0:1789760793.898566:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.898572:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.898581:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28078080, 28082176) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.898591:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28078080=0x1ac7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.898631:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.898634:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28082176 00000020:00010000:2.0:1789760793.898640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.898643:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.898646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.898649:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.898652:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.898658:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.898663:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.898703:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760793.898799:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 28078080<>28082176 00000100:00100000:3.0:1789760793.899091:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.899159:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495961600:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760793.952417:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760793.952427:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.952437:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.952494:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760793.952517:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495961600:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760793.956407:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6855, 6855] 00000080:00200000:2.0:1789760793.956430:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.956436:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.956450:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28082176 i_blocks 55305 00000020:00200000:2.0:1789760793.956455:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.956468:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.956493:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28082176, count: 4096 00000080:00200000:2.0:1789760793.956499:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28082176, bytes: 4096 00000080:00200000:2.0:1789760793.956509:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.956517:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.956519:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28082176, 28086272) 0 0 00000080:00200000:2.0:1789760793.956523:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.956535:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6856, 6856] 00020000:00200000:2.0:1789760793.956542:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28082176 chunk: [28082176, 28086272) 28086272, 4096 00020000:00200000:2.0:1789760793.956547:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.957438:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6856, 6856] 00000008:00010000:2.0:1789760793.957456:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.957462:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760793.957471:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28082176, 28086272) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.957487:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28082176=0x1ac8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.957540:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.957543:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28086272 00000020:00010000:2.0:1789760793.957550:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.957552:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.957555:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.957567:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.957571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.957578:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.957584:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.957607:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760793.957708:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 28082176<>28086272 00000100:00100000:2.0:1789760793.957757:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760793.957777:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495961728:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760794.003747:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974173 gen 2 00000008:00010000:2.0:1789760794.003793:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:2.0:1789760794.003800:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.003808:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.003830:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:2.0:1789760794.003851:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495961728:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760794.003944:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6856, 6856] 00000080:00200000:0.0:1789760794.003957:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.003961:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.003973:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28086272 i_blocks 55305 00000020:00200000:0.0:1789760794.003976:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.003984:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.004020:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28086272, count: 4096 00000080:00200000:0.0:1789760794.004026:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28086272, bytes: 4096 00000080:00200000:0.0:1789760794.004034:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.004041:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.004053:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28086272, 28090368) 0 0 00000080:00200000:0.0:1789760794.004057:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.004070:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6857, 6857] 00020000:00200000:0.0:1789760794.004076:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28086272 chunk: [28086272, 28090368) 28090368, 4096 00020000:00200000:0.0:1789760794.004080:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.004093:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6857, 6857] 00000008:00010000:0.0:1789760794.004107:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.004113:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760794.004121:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28086272, 28090368) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.004134:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28086272=0x1ac9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.004205:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.004207:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28090368 00000020:00010000:0.0:1789760794.004225:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.004228:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.004232:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.004235:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.004238:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.004244:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.004249:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.004264:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760794.004353:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 28086272<>28090368 00000100:00100000:0.0:1789760794.004395:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760794.004408:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495961856:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760794.055062:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974174 gen 2 00000008:00010000:0.0:1789760794.055110:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760794.055118:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.055129:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.055245:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760794.055273:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495961856:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.055408:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6857, 6857] 00000080:00200000:2.0:1789760794.055424:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.055430:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.055446:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28090368 i_blocks 55305 00000020:00200000:2.0:1789760794.055449:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.055459:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.055493:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28090368, count: 4096 00000080:00200000:2.0:1789760794.055501:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28090368, bytes: 4096 00000080:00200000:2.0:1789760794.055510:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.055520:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.055522:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28090368, 28094464) 0 0 00000080:00200000:2.0:1789760794.055527:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.055605:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6858, 6858] 00020000:00200000:2.0:1789760794.055614:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28090368 chunk: [28090368, 28094464) 28094464, 4096 00020000:00200000:2.0:1789760794.055618:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.055637:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6858, 6858] 00000008:00010000:2.0:1789760794.055652:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.055658:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.055669:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28090368, 28094464) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.055683:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28090368=0x1aca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.055743:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.055746:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28094464 00000020:00010000:2.0:1789760794.055754:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.055757:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.055760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.055763:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.055766:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.055773:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.055779:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.055794:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760794.055910:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 28090368<>28094464 00000100:00100000:3.0:1789760794.055987:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495961984:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760794.060951:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760794.108362:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974175 gen 2 00000008:00010000:3.0:1789760794.108413:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760794.108422:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.108435:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.108516:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760794.108545:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495961984:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.108672:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6858, 6858] 00000080:00200000:2.0:1789760794.108689:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.108695:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.108709:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28094464 i_blocks 55305 00000020:00200000:2.0:1789760794.108713:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.108722:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.108754:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28094464, count: 4096 00000080:00200000:2.0:1789760794.108761:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28094464, bytes: 4096 00000080:00200000:2.0:1789760794.108771:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.108779:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.108781:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28094464, 28098560) 0 0 00000080:00200000:2.0:1789760794.108905:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.108923:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6859, 6859] 00020000:00200000:2.0:1789760794.108932:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28094464 chunk: [28094464, 28098560) 28098560, 4096 00020000:00200000:2.0:1789760794.108936:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.108959:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6859, 6859] 00000008:00010000:2.0:1789760794.108976:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.108983:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.108994:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28094464, 28098560) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.109009:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28094464=0x1acb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.109068:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.109071:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28098560 00000020:00010000:2.0:1789760794.109080:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.109084:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.109088:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.109091:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.109095:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.109102:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.109108:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.109127:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760794.110742:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 28094464<>28098560 00000100:00100000:3.0:1789760794.110862:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962112:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760794.179946:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760794.179962:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.179975:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.180066:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760794.180098:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962112:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760794.180196:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6859, 6859] 00000080:00200000:0.0:1789760794.180221:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.180228:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.180243:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28098560 i_blocks 55305 00000020:00200000:0.0:1789760794.180247:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.180256:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.180289:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28098560, count: 4096 00000080:00200000:0.0:1789760794.180361:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28098560, bytes: 4096 00000080:00200000:0.0:1789760794.180378:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.180387:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.180390:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28098560, 28102656) 0 0 00000080:00200000:0.0:1789760794.180394:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.180411:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6860, 6860] 00020000:00200000:0.0:1789760794.180420:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28098560 chunk: [28098560, 28102656) 28102656, 4096 00020000:00200000:0.0:1789760794.180425:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.180446:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6860, 6860] 00000008:00010000:0.0:1789760794.180465:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.180470:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760794.180482:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28098560, 28102656) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.180495:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28098560=0x1acc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.180549:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.180552:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28102656 00000020:00010000:0.0:1789760794.180560:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.180564:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.180567:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.180571:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.180574:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.180582:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.180588:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.180604:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760794.180726:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 28098560<>28102656 00000100:00100000:1.0:1789760794.181559:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495962240:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760794.216996:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974177 gen 2 00000008:00010000:0.0:1789760794.217062:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760794.217072:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.217082:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.217330:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760794.217360:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495962240:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.217508:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6860, 6860] 00000080:00200000:2.0:1789760794.217524:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.217530:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.217610:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28102656 i_blocks 55305 00000020:00200000:2.0:1789760794.217615:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.217625:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.217660:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28102656, count: 4096 00000080:00200000:2.0:1789760794.217668:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28102656, bytes: 4096 00000080:00200000:2.0:1789760794.217682:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.217691:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.217693:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28102656, 28106752) 0 0 00000080:00200000:2.0:1789760794.217698:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.217717:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6861, 6861] 00020000:00200000:2.0:1789760794.217726:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28102656 chunk: [28102656, 28106752) 28106752, 4096 00020000:00200000:2.0:1789760794.217731:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.217753:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6861, 6861] 00000008:00010000:2.0:1789760794.217772:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.217778:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.217789:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28102656, 28106752) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.217813:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28102656=0x1acd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.217999:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.218002:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28106752 00000020:00010000:2.0:1789760794.218011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.218014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.218017:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.218020:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814128 (ffff8dd205fe0248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.218023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.218031:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.218039:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.218056:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760794.218365:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 28102656<>28106752 00000100:00100000:2.0:1789760794.218418:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760794.218436:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962368:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760794.248427:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974178 gen 2 00000008:00010000:2.0:1789760794.248480:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760794.248488:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.248499:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.248527:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760794.248555:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962368:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760794.248631:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6861, 6861] 00000080:00200000:3.0:1789760794.248649:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.248656:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.248673:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28106752 i_blocks 55305 00000020:00200000:3.0:1789760794.248678:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.248688:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.248751:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28106752, count: 4096 00000080:00200000:3.0:1789760794.248760:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28106752, bytes: 4096 00000080:00200000:3.0:1789760794.248771:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.248780:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760794.248782:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28106752, 28110848) 0 0 00000080:00200000:3.0:1789760794.248787:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.248802:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6862, 6862] 00020000:00200000:3.0:1789760794.248811:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28106752 chunk: [28106752, 28110848) 28110848, 4096 00020000:00200000:3.0:1789760794.248816:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.248836:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6862, 6862] 00000008:00010000:3.0:1789760794.248853:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.248858:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760794.248871:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28106752, 28110848) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.248885:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28106752=0x1ace000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.248956:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.248959:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28110848 00000020:00010000:3.0:1789760794.248968:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.248972:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.248975:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.248978:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.248981:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.248988:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.248995:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.249013:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082500c0 00000008:00100000:3.0:1789760794.249140:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 28106752<>28110848 00000100:00100000:2.0:1789760794.249312:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962496:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760794.280453:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082500c0 00000020:00200000:2.0:1789760794.280469:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.280482:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.280499:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082500c0 00000100:00100000:2.0:1789760794.280533:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962496:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760794.280541:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6862, 6862] 00000080:00200000:3.0:1789760794.280562:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.280568:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.280584:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28110848 i_blocks 55305 00000020:00200000:3.0:1789760794.280590:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.280680:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.280714:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28110848, count: 4096 00000080:00200000:2.0:1789760794.280721:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28110848, bytes: 4096 00000080:00200000:2.0:1789760794.280732:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.280741:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.280743:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28110848, 28114944) 0 0 00000080:00200000:2.0:1789760794.280748:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.280764:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6863, 6863] 00020000:00200000:2.0:1789760794.280773:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28110848 chunk: [28110848, 28114944) 28114944, 4096 00020000:00200000:2.0:1789760794.280779:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.280800:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6863, 6863] 00000008:00010000:2.0:1789760794.280816:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.280822:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.280834:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28110848, 28114944) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.280848:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28110848=0x1acf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.280901:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.280904:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28114944 00000020:00010000:2.0:1789760794.280913:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.280917:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.280920:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.280924:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.280927:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.280935:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.280954:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.280996:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760794.281123:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 28110848<>28114944 00000100:00100000:2.0:1789760794.281162:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760794.281220:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962624:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760794.332053:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974180 gen 2 00000008:00010000:2.0:1789760794.332115:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760794.332123:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.332134:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.332256:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760794.332284:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962624:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760794.333392:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6863, 6863] 00000080:00200000:0.0:1789760794.333412:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.333420:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.333437:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28114944 i_blocks 55305 00000020:00200000:0.0:1789760794.333441:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.333452:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.333487:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28114944, count: 4096 00000080:00200000:0.0:1789760794.333496:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28114944, bytes: 4096 00000080:00200000:0.0:1789760794.333508:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.333517:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.333519:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28114944, 28119040) 0 0 00000080:00200000:0.0:1789760794.333523:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.333541:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6864, 6864] 00020000:00200000:0.0:1789760794.333550:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28114944 chunk: [28114944, 28119040) 28119040, 4096 00020000:00200000:0.0:1789760794.333554:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.333575:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6864, 6864] 00000008:00010000:0.0:1789760794.333593:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.333598:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760794.333609:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28114944, 28119040) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.333623:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28114944=0x1ad0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.333688:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.333691:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28119040 00000020:00010000:0.0:1789760794.333699:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.333703:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.333706:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.333715:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.333718:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.333725:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.333733:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.333756:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760794.333880:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 28114944<>28119040 00000100:00100000:1.0:1789760794.333952:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495962752:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760794.387424:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974181 gen 2 00000008:00010000:0.0:1789760794.387547:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760794.387555:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.387566:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.387586:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760794.387618:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495962752:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.387718:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6864, 6864] 00000080:00200000:2.0:1789760794.387736:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.387742:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.387758:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28119040 i_blocks 55305 00000020:00200000:2.0:1789760794.387762:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.387772:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.387807:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28119040, count: 4096 00000080:00200000:2.0:1789760794.387814:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28119040, bytes: 4096 00000080:00200000:2.0:1789760794.387826:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.387836:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.387838:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28119040, 28123136) 0 0 00000080:00200000:2.0:1789760794.387842:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.387860:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6865, 6865] 00020000:00200000:2.0:1789760794.387869:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28119040 chunk: [28119040, 28123136) 28123136, 4096 00020000:00200000:2.0:1789760794.387873:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.387892:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6865, 6865] 00000008:00010000:2.0:1789760794.387908:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.387914:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.387925:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28119040, 28123136) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.387939:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28119040=0x1ad1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.387993:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.387997:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28123136 00000020:00010000:2.0:1789760794.388005:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.388008:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.388011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.388014:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.388017:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.388024:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.388031:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.388046:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760794.388221:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 28119040<>28123136 00000100:00100000:2.0:1789760794.388268:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962880:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760794.421933:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:2.0:1789760794.421949:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.421960:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.422307:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:2.0:1789760794.422340:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495962880:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760794.422481:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6865, 6865] 00000080:00200000:0.0:1789760794.422498:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.422504:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.422524:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28123136 i_blocks 55305 00000020:00200000:0.0:1789760794.422527:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.422538:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.422574:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28123136, count: 4096 00000080:00200000:0.0:1789760794.422582:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28123136, bytes: 4096 00000080:00200000:0.0:1789760794.422598:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.422607:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.422610:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28123136, 28127232) 0 0 00000080:00200000:0.0:1789760794.422615:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.422633:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6866, 6866] 00020000:00200000:0.0:1789760794.422642:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28123136 chunk: [28123136, 28127232) 28127232, 4096 00020000:00200000:0.0:1789760794.422646:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.422666:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6866, 6866] 00000008:00010000:0.0:1789760794.422698:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.422703:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760794.422714:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28123136, 28127232) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.422728:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28123136=0x1ad2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.422787:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.422790:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28127232 00000020:00010000:0.0:1789760794.422799:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.422802:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.422806:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.422810:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.422812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.422820:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.422826:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.422843:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760794.422965:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 28123136<>28127232 00000100:00100000:0.0:1789760794.423015:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760794.423032:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495963008:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760794.450459:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974183 gen 2 00000008:00010000:0.0:1789760794.450532:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760794.450542:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.450555:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.450578:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760794.450619:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495963008:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760794.450828:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6866, 6866] 00000080:00200000:1.0:1789760794.450845:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.450852:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.450868:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28127232 i_blocks 55305 00000020:00200000:1.0:1789760794.450872:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.450882:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.450916:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28127232, count: 4096 00000080:00200000:1.0:1789760794.450923:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28127232, bytes: 4096 00000080:00200000:1.0:1789760794.450934:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.450943:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760794.450945:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28127232, 28131328) 0 0 00000080:00200000:1.0:1789760794.450950:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.450967:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6867, 6867] 00020000:00200000:1.0:1789760794.450975:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28127232 chunk: [28127232, 28131328) 28131328, 4096 00020000:00200000:1.0:1789760794.450980:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.451000:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6867, 6867] 00000008:00010000:1.0:1789760794.451018:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.451023:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760794.451036:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28127232, 28131328) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.451052:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28127232=0x1ad3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.451109:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.451113:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28131328 00000020:00010000:1.0:1789760794.451121:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.451125:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.451128:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.451132:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.451134:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.451142:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.451149:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.451166:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760794.451463:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 28127232<>28131328 00000100:00100000:1.0:1789760794.451497:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.451514:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495963136:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760794.508759:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495963392:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760794.508912:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1789760794.508921:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2358:1876699495963264:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760794.508956:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495963520:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760794.510793:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974184 gen 2 00000008:00010000:0.0:1789760794.510845:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760794.510922:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.510934:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.511037:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760794.511068:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495963136:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760794.511295:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6867, 6867] 00000080:00200000:1.0:1789760794.511379:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.511387:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.511403:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28131328 i_blocks 55305 00000020:00200000:1.0:1789760794.511407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.511417:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.511451:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28131328, count: 4096 00000080:00200000:1.0:1789760794.511458:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28131328, bytes: 4096 00000080:00200000:1.0:1789760794.511471:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.511480:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760794.511483:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28131328, 28135424) 0 0 00000080:00200000:1.0:1789760794.511488:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.511504:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6868, 6868] 00020000:00200000:1.0:1789760794.511514:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28131328 chunk: [28131328, 28135424) 28135424, 4096 00020000:00200000:1.0:1789760794.511519:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.511541:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6868, 6868] 00000008:00010000:1.0:1789760794.511560:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.511567:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760794.511578:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28131328, 28135424) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.511594:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28131328=0x1ad4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.511653:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.511655:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28135424 00000020:00010000:1.0:1789760794.511664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.511667:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.511670:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.511674:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.511677:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.511684:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.511691:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.511709:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760794.511853:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 28131328<>28135424 00000100:00100000:0.0:1789760794.511931:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495963648:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760794.514044:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495963392:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760794.514597:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5292164f-12cc-4fea-b062-6f9c12f1be43:2358:1876699495963264:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760794.514613:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495963520:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760794.557131:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974185 gen 2 00000008:00010000:0.0:1789760794.557248:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760794.557257:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.557267:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.557281:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760794.557309:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495963648:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760794.557373:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6868, 6868] 00000080:00200000:1.0:1789760794.557391:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.557397:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.557413:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28135424 i_blocks 55305 00000020:00200000:1.0:1789760794.557417:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.557426:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.557466:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28135424, count: 4096 00000080:00200000:1.0:1789760794.557473:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28135424, bytes: 4096 00000080:00200000:1.0:1789760794.557483:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.557492:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760794.557494:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28135424, 28139520) 0 0 00000080:00200000:1.0:1789760794.557498:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.557515:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6869, 6869] 00020000:00200000:1.0:1789760794.557524:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28135424 chunk: [28135424, 28139520) 28139520, 4096 00020000:00200000:1.0:1789760794.557529:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.557555:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6869, 6869] 00000008:00010000:1.0:1789760794.557574:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.557580:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760794.557591:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28135424, 28139520) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.557607:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28135424=0x1ad5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.557663:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.557665:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28139520 00000020:00010000:1.0:1789760794.557675:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.557679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.557682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.557685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.557688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.557695:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.557703:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.557720:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760794.557901:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 28135424<>28139520 00000100:00100000:0.0:1789760794.557985:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495963776:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760794.589636:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760794.589653:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.589665:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.589726:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760794.589761:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495963776:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760794.589834:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6869, 6869] 00000080:00200000:1.0:1789760794.589903:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760794.589911:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760794.589927:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28139520 i_blocks 55305 00000020:00200000:1.0:1789760794.589931:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760794.589941:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760794.589978:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28139520, count: 4096 00000080:00200000:1.0:1789760794.589985:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28139520, bytes: 4096 00000080:00200000:1.0:1789760794.589997:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760794.590007:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760794.590010:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28139520, 28143616) 0 0 00000080:00200000:1.0:1789760794.590015:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760794.590033:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6870, 6870] 00020000:00200000:1.0:1789760794.590042:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28139520 chunk: [28139520, 28143616) 28143616, 4096 00020000:00200000:1.0:1789760794.590047:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760794.590071:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6870, 6870] 00000008:00010000:1.0:1789760794.590090:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760794.590096:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760794.590108:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28139520, 28143616) append 0 append lockpos 0 00000080:00200000:1.0:1789760794.590125:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28139520=0x1ad6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760794.590348:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760794.590351:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28143616 00000020:00010000:1.0:1789760794.590361:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760794.590365:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.590368:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760794.590371:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760794.590373:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760794.590380:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760794.590388:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760794.590406:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760794.590534:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 28139520<>28143616 00000100:00100000:0.0:1789760794.590607:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495963904:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760794.626270:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974186 gen 2 00000100:00100000:0.0:1789760794.626322:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d500 x1876699495963904/t4294974187(4294974187) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760810 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760794.626372:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760794.626379:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760794.626389:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760794.626413:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760794.626439:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495963904:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.626640:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6870, 6870] 00000080:00200000:2.0:1789760794.626658:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.626664:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.626679:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28143616 i_blocks 55305 00000020:00200000:2.0:1789760794.626683:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.626693:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.626729:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28143616, count: 4096 00000080:00200000:2.0:1789760794.626736:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28143616, bytes: 4096 00000080:00200000:2.0:1789760794.626747:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.626756:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.626758:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28143616, 28147712) 0 0 00000080:00200000:2.0:1789760794.626763:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.626779:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6871, 6871] 00020000:00200000:2.0:1789760794.626787:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28143616 chunk: [28143616, 28147712) 28147712, 4096 00020000:00200000:2.0:1789760794.626791:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.626811:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6871, 6871] 00000008:00010000:2.0:1789760794.626828:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.626833:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.626846:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28143616, 28147712) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.626872:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28143616=0x1ad7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.626925:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.626928:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28147712 00000020:00010000:2.0:1789760794.626939:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.626942:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.626946:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.626950:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.626954:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.626962:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.626969:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.626984:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760794.627103:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 28143616<>28147712 00000100:00100000:3.0:1789760794.627435:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495964032:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760794.685533:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974188 gen 2 00000008:00010000:3.0:1789760794.685613:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760794.685623:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.685634:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.685659:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760794.685687:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495964032:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.685800:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6871, 6871] 00000080:00200000:2.0:1789760794.685819:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.685825:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.685842:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28147712 i_blocks 55305 00000020:00200000:2.0:1789760794.685845:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.685855:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.685893:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28147712, count: 4096 00000080:00200000:2.0:1789760794.685900:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28147712, bytes: 4096 00000080:00200000:2.0:1789760794.685911:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.685921:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.685923:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28147712, 28151808) 0 0 00000080:00200000:2.0:1789760794.685929:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.685950:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6872, 6872] 00020000:00200000:2.0:1789760794.685959:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28147712 chunk: [28147712, 28151808) 28151808, 4096 00020000:00200000:2.0:1789760794.685963:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.685982:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6872, 6872] 00000008:00010000:2.0:1789760794.686000:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.686006:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.686018:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28147712, 28151808) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.686033:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28147712=0x1ad8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.686087:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.686089:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28151808 00000020:00010000:2.0:1789760794.686099:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.686102:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.686105:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.686108:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.686111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.686119:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.686126:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.686142:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760794.686344:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 28147712<>28151808 00000100:00100000:3.0:1789760794.686637:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760794.686655:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495964160:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760794.724818:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974189 gen 2 00000008:00010000:3.0:1789760794.724967:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760794.724976:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.724987:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.725140:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760794.725165:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495964160:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.725258:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6872, 6872] 00000080:00200000:2.0:1789760794.725280:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.725287:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.725312:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28151808 i_blocks 55305 00000020:00200000:2.0:1789760794.725316:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.725325:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.725364:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28151808, count: 4096 00000080:00200000:2.0:1789760794.725372:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28151808, bytes: 4096 00000080:00200000:2.0:1789760794.725384:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.725393:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.725396:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28151808, 28155904) 0 0 00000080:00200000:2.0:1789760794.725501:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.725520:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6873, 6873] 00020000:00200000:2.0:1789760794.725529:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28151808 chunk: [28151808, 28155904) 28155904, 4096 00020000:00200000:2.0:1789760794.725533:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.725553:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6873, 6873] 00000008:00010000:2.0:1789760794.725571:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.725576:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.725588:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28151808, 28155904) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.725602:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28151808=0x1ad9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.725663:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.725665:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28155904 00000020:00010000:2.0:1789760794.725674:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.725676:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.725678:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.725681:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.725683:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.725688:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.725693:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.725706:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760794.725963:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 28151808<>28155904 00000100:00100000:3.0:1789760794.726081:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495964288:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760794.776893:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974190 gen 2 00000008:00010000:3.0:1789760794.776944:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760794.776953:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.776964:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.776991:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760794.777020:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495964288:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.777139:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6873, 6873] 00000080:00200000:2.0:1789760794.777156:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.777163:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.777218:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28155904 i_blocks 55305 00000020:00200000:2.0:1789760794.777223:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.777233:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.777266:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28155904, count: 4096 00000080:00200000:2.0:1789760794.777273:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28155904, bytes: 4096 00000080:00200000:2.0:1789760794.777285:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.777294:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.777296:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28155904, 28160000) 0 0 00000080:00200000:2.0:1789760794.777301:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.777317:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6874, 6874] 00020000:00200000:2.0:1789760794.777326:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28155904 chunk: [28155904, 28160000) 28160000, 4096 00020000:00200000:2.0:1789760794.777331:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.777349:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6874, 6874] 00000008:00010000:2.0:1789760794.777367:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.777373:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.777384:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28155904, 28160000) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.777397:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28155904=0x1ada000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.777453:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.777455:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28160000 00000020:00010000:2.0:1789760794.777465:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.777468:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.777471:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.777475:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.777478:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.777484:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.777491:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.777508:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760794.777626:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 28155904<>28160000 00000100:00100000:3.0:1789760794.778998:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760794.779029:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495964416:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760794.840809:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974191 gen 2 00000008:00010000:3.0:1789760794.840863:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760794.840871:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.841108:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.841231:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760794.841260:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495964416:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.841702:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6874, 6874] 00000080:00200000:2.0:1789760794.841778:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.841786:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.841803:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28160000 i_blocks 55305 00000020:00200000:2.0:1789760794.841806:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.841817:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.841852:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28160000, count: 4096 00000080:00200000:2.0:1789760794.841859:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28160000, bytes: 4096 00000080:00200000:2.0:1789760794.841874:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.841883:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.841885:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28160000, 28164096) 0 0 00000080:00200000:2.0:1789760794.841890:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.841909:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6875, 6875] 00020000:00200000:2.0:1789760794.841917:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28160000 chunk: [28160000, 28164096) 28164096, 4096 00020000:00200000:2.0:1789760794.841922:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.841944:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6875, 6875] 00000008:00010000:2.0:1789760794.842667:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.842677:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.842694:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28160000, 28164096) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.842711:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28160000=0x1adb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.842795:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.842798:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28164096 00000020:00010000:2.0:1789760794.842807:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.842811:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.842814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.842818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.842821:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.842831:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.842839:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.842859:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760794.842994:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 28160000<>28164096 00000100:00100000:3.0:1789760794.843300:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760794.843328:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495964544:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760794.895705:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974192 gen 2 00000008:00010000:3.0:1789760794.895766:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760794.895775:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.895785:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.895811:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760794.895842:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495964544:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760794.896139:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6875, 6875] 00000080:00200000:2.0:1789760794.896156:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.896162:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.896224:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28164096 i_blocks 55305 00000020:00200000:2.0:1789760794.896228:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.896237:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.896277:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28164096, count: 4096 00000080:00200000:2.0:1789760794.896286:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28164096, bytes: 4096 00000080:00200000:2.0:1789760794.896298:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.896310:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.896313:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28164096, 28168192) 0 0 00000080:00200000:2.0:1789760794.896317:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.896333:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6876, 6876] 00020000:00200000:2.0:1789760794.896342:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28164096 chunk: [28164096, 28168192) 28168192, 4096 00020000:00200000:2.0:1789760794.896347:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.896368:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6876, 6876] 00000008:00010000:2.0:1789760794.896386:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.896392:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760794.896405:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28164096, 28168192) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.896420:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28164096=0x1adc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.896477:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.896479:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28168192 00000020:00010000:2.0:1789760794.896487:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.896490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.896494:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.896497:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.896500:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.896507:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.896513:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.896700:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760794.896949:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 28164096<>28168192 00000100:00100000:3.0:1789760794.897599:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495964672:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760794.939656:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760794.939673:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760794.939684:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760794.939784:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760794.939818:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495964672:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760794.939934:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6876, 6876] 00000080:00200000:0.0:1789760794.939952:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.939959:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.939975:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28168192 i_blocks 55305 00000020:00200000:0.0:1789760794.939980:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.939989:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.940023:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28168192, count: 4096 00000080:00200000:0.0:1789760794.940395:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28168192, bytes: 4096 00000080:00200000:0.0:1789760794.940409:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.940419:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.940422:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28168192, 28172288) 0 0 00000080:00200000:0.0:1789760794.940427:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.940444:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6877, 6877] 00020000:00200000:0.0:1789760794.940452:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28168192 chunk: [28168192, 28172288) 28172288, 4096 00020000:00200000:0.0:1789760794.940457:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.940477:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6877, 6877] 00000008:00010000:0.0:1789760794.940496:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.940502:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760794.940514:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28168192, 28172288) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.940598:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28168192=0x1add000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.940664:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.940667:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28172288 00000020:00010000:0.0:1789760794.940675:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.940679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.940682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.940685:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.940688:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.940695:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.940702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.940784:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760794.941738:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 28168192<>28172288 00000100:00100000:0.0:1789760794.941800:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760794.941826:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495964800:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.009281:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974194 gen 2 00000008:00010000:0.0:1789760795.009346:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760795.009355:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.009365:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.009385:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760795.009416:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495964800:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.009643:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6877, 6877] 00000080:00200000:2.0:1789760795.009660:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.009667:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.009685:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28172288 i_blocks 55305 00000020:00200000:2.0:1789760795.009690:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.009701:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.009808:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28172288, count: 4096 00000080:00200000:2.0:1789760795.009816:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28172288, bytes: 4096 00000080:00200000:2.0:1789760795.009827:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.009836:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.009838:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28172288, 28176384) 0 0 00000080:00200000:2.0:1789760795.009842:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.009869:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6878, 6878] 00020000:00200000:2.0:1789760795.009877:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28172288 chunk: [28172288, 28176384) 28176384, 4096 00020000:00200000:2.0:1789760795.009881:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.009901:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6878, 6878] 00000008:00010000:2.0:1789760795.009920:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.009926:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.009938:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28172288, 28176384) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.009956:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28172288=0x1ade000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.010025:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.010028:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28176384 00000020:00010000:2.0:1789760795.010037:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.010040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.010043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.010046:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.010050:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.010057:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.010065:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.010083:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760795.010265:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 28172288<>28176384 00000100:00100000:2.0:1789760795.010317:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760795.010336:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495964928:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760795.069496:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974195 gen 2 00000008:00010000:2.0:1789760795.069557:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760795.069567:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.069577:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.069603:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760795.069632:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495964928:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760795.069914:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6878, 6878] 00000080:00200000:0.0:1789760795.069930:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.069937:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.069953:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28176384 i_blocks 55305 00000020:00200000:0.0:1789760795.069956:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.069967:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.070001:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28176384, count: 4096 00000080:00200000:0.0:1789760795.070008:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28176384, bytes: 4096 00000080:00200000:0.0:1789760795.070018:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.070027:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.070029:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28176384, 28180480) 0 0 00000080:00200000:0.0:1789760795.070034:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.070052:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6879, 6879] 00020000:00200000:0.0:1789760795.070060:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28176384 chunk: [28176384, 28180480) 28180480, 4096 00020000:00200000:0.0:1789760795.070065:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.070085:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6879, 6879] 00000008:00010000:0.0:1789760795.070102:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.070108:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760795.070120:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28176384, 28180480) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.070136:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28176384=0x1adf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.070400:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.070404:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28180480 00000020:00010000:0.0:1789760795.070413:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.070416:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.070419:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.070423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.070426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.070434:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.070440:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.070452:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760795.070603:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211837800 - object 0x240000400:4 offset 28176384<>28180480 00000100:00100000:1.0:1789760795.070677:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495965056:192.168.203.109@tcp:4:dd.0 00010000:00010000:2.0:1789760795.083921:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:0.0:1789760795.109819:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760795.109834:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.109845:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.109958:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6879, 6879] 00000008:00010000:0.0:1789760795.109961:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000080:00200000:2.0:1789760795.109977:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.109983:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:0.0:1789760795.109990:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495965056:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.109998:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28180480 i_blocks 55305 00000020:00200000:2.0:1789760795.110001:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.110011:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.110046:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28180480, count: 4096 00000080:00200000:2.0:1789760795.110053:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28180480, bytes: 4096 00000080:00200000:2.0:1789760795.110063:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.110071:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.110073:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28180480, 28184576) 0 0 00000080:00200000:2.0:1789760795.110077:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.110092:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6880, 6880] 00020000:00200000:2.0:1789760795.110099:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28180480 chunk: [28180480, 28184576) 28184576, 4096 00020000:00200000:2.0:1789760795.110102:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.110120:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6880, 6880] 00000008:00010000:2.0:1789760795.110136:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.110141:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.110151:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28180480, 28184576) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.110162:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28180480=0x1ae0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.110694:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.110699:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28184576 00000020:00010000:2.0:1789760795.110775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.110780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.110784:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.110787:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.110790:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.110798:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.110806:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.110818:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760795.111000:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 28180480<>28184576 00000100:00100000:3.0:1789760795.111070:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495965184:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760795.145907:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974197 gen 2 00000008:00010000:2.0:1789760795.145975:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760795.145983:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.145994:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.146022:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760795.146052:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495965184:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760795.146363:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6880, 6880] 00000080:00200000:0.0:1789760795.146381:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.146387:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.146403:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28184576 i_blocks 55305 00000020:00200000:0.0:1789760795.146407:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.146418:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.146458:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28184576, count: 4096 00000080:00200000:0.0:1789760795.146466:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28184576, bytes: 4096 00000080:00200000:0.0:1789760795.146477:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.146486:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.146488:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28184576, 28188672) 0 0 00000080:00200000:0.0:1789760795.146492:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.146508:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6881, 6881] 00020000:00200000:0.0:1789760795.146517:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28184576 chunk: [28184576, 28188672) 28188672, 4096 00020000:00200000:0.0:1789760795.146522:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.146543:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6881, 6881] 00000008:00010000:0.0:1789760795.146561:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.146568:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760795.146579:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28184576, 28188672) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.146594:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28184576=0x1ae1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.146648:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.146650:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28188672 00000020:00010000:0.0:1789760795.146657:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.146660:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.146662:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.146664:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.146666:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.146672:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.146677:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.146690:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760795.146800:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 28184576<>28188672 00000100:00100000:0.0:1789760795.146840:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760795.146858:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495965312:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760795.176659:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760795.176672:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.176684:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.176736:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760795.176766:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495965312:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.176959:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6881, 6881] 00000080:00200000:1.0:1789760795.176976:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.176982:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.176998:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28188672 i_blocks 55305 00000020:00200000:1.0:1789760795.177001:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.177010:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.177046:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28188672, count: 4096 00000080:00200000:1.0:1789760795.177053:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28188672, bytes: 4096 00000080:00200000:1.0:1789760795.177062:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.177071:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.177073:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28188672, 28192768) 0 0 00000080:00200000:1.0:1789760795.177077:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.177093:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6882, 6882] 00020000:00200000:1.0:1789760795.177101:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28188672 chunk: [28188672, 28192768) 28192768, 4096 00020000:00200000:1.0:1789760795.177105:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.177125:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6882, 6882] 00000008:00010000:1.0:1789760795.177142:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.177147:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.177157:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28188672, 28192768) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.177316:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28188672=0x1ae2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.177375:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.177378:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28192768 00000020:00010000:1.0:1789760795.177387:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.177390:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.177393:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.177397:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.177400:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.177407:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.177414:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.177435:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760795.177623:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 28188672<>28192768 00000100:00100000:0.0:1789760795.177678:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495965440:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.211931:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974199 gen 2 00000008:00010000:0.0:1789760795.212007:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760795.212018:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.212040:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.212061:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760795.212092:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495965440:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.212262:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6882, 6882] 00000080:00200000:1.0:1789760795.212278:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.212285:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.212300:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28192768 i_blocks 55305 00000020:00200000:1.0:1789760795.212304:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.212312:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.212346:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28192768, count: 4096 00000080:00200000:1.0:1789760795.212353:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28192768, bytes: 4096 00000080:00200000:1.0:1789760795.212364:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.212374:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.212376:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28192768, 28196864) 0 0 00000080:00200000:1.0:1789760795.212381:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.212399:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6883, 6883] 00020000:00200000:1.0:1789760795.212408:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28192768 chunk: [28192768, 28196864) 28196864, 4096 00020000:00200000:1.0:1789760795.212413:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.212435:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6883, 6883] 00000008:00010000:1.0:1789760795.212454:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.212460:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.212472:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28192768, 28196864) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.212486:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28192768=0x1ae3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.212619:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.212622:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28196864 00000020:00010000:1.0:1789760795.212631:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.212634:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.212637:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.212640:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.212643:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.212651:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.212658:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.212675:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760795.212797:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 28192768<>28196864 00000100:00100000:0.0:1789760795.213063:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495965568:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.259698:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974200 gen 2 00000008:00010000:0.0:1789760795.259750:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760795.259759:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.259770:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.259792:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760795.259821:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495965568:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.259975:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6883, 6883] 00000080:00200000:1.0:1789760795.259991:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.259998:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.260013:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28196864 i_blocks 55305 00000020:00200000:1.0:1789760795.260017:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.260026:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.260068:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28196864, count: 4096 00000080:00200000:1.0:1789760795.260075:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28196864, bytes: 4096 00000080:00200000:1.0:1789760795.260085:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.260093:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.260096:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28196864, 28200960) 0 0 00000080:00200000:1.0:1789760795.260100:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.260116:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6884, 6884] 00020000:00200000:1.0:1789760795.260124:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28196864 chunk: [28196864, 28200960) 28200960, 4096 00020000:00200000:1.0:1789760795.260129:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.260148:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6884, 6884] 00000008:00010000:1.0:1789760795.260164:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.260330:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.260344:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28196864, 28200960) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.260358:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28196864=0x1ae4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.260415:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.260418:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28200960 00000020:00010000:1.0:1789760795.260427:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.260431:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.260435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.260438:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.260441:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.260449:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.260456:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.260472:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760795.260829:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 28196864<>28200960 00000100:00100000:0.0:1789760795.261001:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760795.261021:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495965696:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.299883:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974201 gen 2 00000008:00010000:0.0:1789760795.299936:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760795.299945:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.299957:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.299979:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760795.300008:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495965696:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.300120:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6884, 6884] 00000080:00200000:1.0:1789760795.300138:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.300144:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.300160:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28200960 i_blocks 55305 00000020:00200000:1.0:1789760795.300163:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.300196:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.300239:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28200960, count: 4096 00000080:00200000:1.0:1789760795.300247:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28200960, bytes: 4096 00000080:00200000:1.0:1789760795.300258:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.300267:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.300269:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28200960, 28205056) 0 0 00000080:00200000:1.0:1789760795.300274:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.300289:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6885, 6885] 00020000:00200000:1.0:1789760795.300297:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28200960 chunk: [28200960, 28205056) 28205056, 4096 00020000:00200000:1.0:1789760795.300301:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.300320:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6885, 6885] 00000008:00010000:1.0:1789760795.300339:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.300345:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.300357:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28200960, 28205056) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.300372:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28200960=0x1ae5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.300426:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.300428:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28205056 00000020:00010000:1.0:1789760795.300437:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.300440:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.300443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.300447:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.300450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.300457:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.300464:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.300570:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:0.0:1789760795.300695:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 28200960<>28205056 00000100:00100000:0.0:1789760795.300736:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495965824:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760795.356159:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760795.356191:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.356213:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.356241:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760795.356277:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495965824:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.356520:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6885, 6885] 00000080:00200000:1.0:1789760795.356536:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.356543:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.356558:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28205056 i_blocks 55305 00000020:00200000:1.0:1789760795.356561:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.356571:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.356612:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28205056, count: 4096 00000080:00200000:1.0:1789760795.356619:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28205056, bytes: 4096 00000080:00200000:1.0:1789760795.356634:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.356646:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.356648:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28205056, 28209152) 0 0 00000080:00200000:1.0:1789760795.356653:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.356668:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6886, 6886] 00020000:00200000:1.0:1789760795.356676:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28205056 chunk: [28205056, 28209152) 28209152, 4096 00020000:00200000:1.0:1789760795.356681:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.356702:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6886, 6886] 00000008:00010000:1.0:1789760795.356721:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.356726:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.356737:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28205056, 28209152) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.356764:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28205056=0x1ae6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.356821:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.356824:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28209152 00000020:00010000:1.0:1789760795.356833:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.356836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.356838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.356841:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.356844:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.356851:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.356857:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.356874:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760795.356998:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 28205056<>28209152 00000100:00100000:0.0:1789760795.357305:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495965952:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.387570:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974203 gen 2 00000008:00010000:0.0:1789760795.387633:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760795.387642:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.387652:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.387674:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760795.387705:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495965952:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.387859:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6886, 6886] 00000080:00200000:1.0:1789760795.387875:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.387881:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.387897:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28209152 i_blocks 55305 00000020:00200000:1.0:1789760795.387900:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.387909:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.387943:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28209152, count: 4096 00000080:00200000:1.0:1789760795.387951:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28209152, bytes: 4096 00000080:00200000:1.0:1789760795.387961:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.387970:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.387972:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28209152, 28213248) 0 0 00000080:00200000:1.0:1789760795.387977:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.387993:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6887, 6887] 00020000:00200000:1.0:1789760795.388002:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28209152 chunk: [28209152, 28213248) 28213248, 4096 00020000:00200000:1.0:1789760795.388007:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.388036:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6887, 6887] 00000008:00010000:1.0:1789760795.388055:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.388062:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.388073:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28209152, 28213248) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.388088:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28209152=0x1ae7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.388146:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.388149:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28213248 00000020:00010000:1.0:1789760795.388157:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.388161:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.388164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.388198:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.388202:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.388219:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.388225:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.388242:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760795.388365:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 28209152<>28213248 00000100:00100000:0.0:1789760795.389670:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495966080:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.423310:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974204 gen 2 00000008:00010000:1.0:1789760795.423441:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:1.0:1789760795.423451:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.423463:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.423600:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:1.0:1789760795.423633:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495966080:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.423710:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6887, 6887] 00000080:00200000:2.0:1789760795.423730:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.423738:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.423755:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28213248 i_blocks 55305 00000020:00200000:2.0:1789760795.423758:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.423768:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.423804:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28213248, count: 4096 00000080:00200000:2.0:1789760795.423811:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28213248, bytes: 4096 00000080:00200000:2.0:1789760795.423822:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.423832:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.423835:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28213248, 28217344) 0 0 00000080:00200000:2.0:1789760795.423839:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.423859:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6888, 6888] 00020000:00200000:2.0:1789760795.423868:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28213248 chunk: [28213248, 28217344) 28217344, 4096 00020000:00200000:2.0:1789760795.423874:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.423897:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6888, 6888] 00000008:00010000:2.0:1789760795.423917:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.423923:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.423935:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28213248, 28217344) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.423950:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28213248=0x1ae8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.424004:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.424007:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28217344 00000020:00010000:2.0:1789760795.424016:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.424020:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.424023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.424027:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.424030:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.424038:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.424045:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.424062:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760795.424220:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 28213248<>28217344 00000100:00100000:2.0:1789760795.424266:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760795.424282:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495966208:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760795.458262:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974205 gen 2 00000008:00010000:2.0:1789760795.458314:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:2.0:1789760795.458322:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.458333:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.458357:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:2.0:1789760795.458387:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495966208:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.458491:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6888, 6888] 00000080:00200000:1.0:1789760795.458507:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.458513:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.458538:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28217344 i_blocks 55305 00000020:00200000:1.0:1789760795.458542:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.458552:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.458587:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28217344, count: 4096 00000080:00200000:1.0:1789760795.458593:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28217344, bytes: 4096 00000080:00200000:1.0:1789760795.458603:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.458611:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.458613:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28217344, 28221440) 0 0 00000080:00200000:1.0:1789760795.458618:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.458634:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6889, 6889] 00020000:00200000:1.0:1789760795.458643:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28217344 chunk: [28217344, 28221440) 28221440, 4096 00020000:00200000:1.0:1789760795.458647:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.458666:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6889, 6889] 00000008:00010000:1.0:1789760795.458684:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.458689:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.458701:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28217344, 28221440) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.458714:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28217344=0x1ae9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.458766:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.458768:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28221440 00000020:00010000:1.0:1789760795.458776:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.458779:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.458781:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.458783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.458785:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.458790:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.458794:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.458806:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79600 00000008:00100000:1.0:1789760795.459036:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 28217344<>28221440 00000100:00100000:0.0:1789760795.459141:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495966336:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760795.493309:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79600 00000020:00200000:0.0:1789760795.493326:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.493338:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.493717:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79600 00000100:00100000:0.0:1789760795.493753:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495966336:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.494042:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6889, 6889] 00000080:00200000:2.0:1789760795.499663:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.499680:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.499701:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28221440 i_blocks 55305 00000020:00200000:2.0:1789760795.499707:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.499727:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.500012:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28221440, count: 4096 00000080:00200000:2.0:1789760795.500023:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28221440, bytes: 4096 00000080:00200000:2.0:1789760795.500038:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.500046:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.500049:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28221440, 28225536) 0 0 00000080:00200000:2.0:1789760795.500053:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.500069:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6890, 6890] 00020000:00200000:2.0:1789760795.500077:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28221440 chunk: [28221440, 28225536) 28225536, 4096 00020000:00200000:2.0:1789760795.500082:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.500101:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6890, 6890] 00000008:00010000:2.0:1789760795.500129:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.500134:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.500146:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28221440, 28225536) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.500159:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28221440=0x1aea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.500721:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.500727:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28225536 00000020:00010000:2.0:1789760795.500737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.500741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.501568:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.501574:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.501576:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.501591:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.501730:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.502902:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760795.503147:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265180 - object 0x240000400:4 offset 28221440<>28225536 00000100:00100000:3.0:1789760795.503296:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495966464:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760795.548320:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974207 gen 2 00000008:00010000:3.0:1789760795.548387:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760795.548396:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.548407:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.548516:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760795.548550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495966464:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.548659:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6890, 6890] 00000080:00200000:2.0:1789760795.548673:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.548678:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.548692:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28225536 i_blocks 55305 00000020:00200000:2.0:1789760795.548695:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.548703:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.548733:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28225536, count: 4096 00000080:00200000:2.0:1789760795.548738:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28225536, bytes: 4096 00000080:00200000:2.0:1789760795.548746:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.548753:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.548755:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28225536, 28229632) 0 0 00000080:00200000:2.0:1789760795.548758:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.548772:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6891, 6891] 00020000:00200000:2.0:1789760795.548779:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28225536 chunk: [28225536, 28229632) 28229632, 4096 00020000:00200000:2.0:1789760795.548783:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.548799:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6891, 6891] 00000008:00010000:2.0:1789760795.548813:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.548818:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.548828:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28225536, 28229632) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.548841:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28225536=0x1aeb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.548961:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.548964:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28229632 00000020:00010000:2.0:1789760795.548973:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.548977:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.548980:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.548983:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.548987:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.548994:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.549002:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.549020:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760795.549149:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265f80 - object 0x240000400:4 offset 28225536<>28229632 00000100:00100000:3.0:1789760795.549237:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495966592:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760795.597670:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974208 gen 2 00000008:00010000:3.0:1789760795.597721:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760795.597729:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.597741:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.597767:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760795.597797:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495966592:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.598165:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6891, 6891] 00000080:00200000:2.0:1789760795.598294:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.598302:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.598319:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28229632 i_blocks 55305 00000020:00200000:2.0:1789760795.598322:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.598333:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.598369:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28229632, count: 4096 00000080:00200000:2.0:1789760795.598377:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28229632, bytes: 4096 00000080:00200000:2.0:1789760795.598388:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.598397:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.598399:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28229632, 28233728) 0 0 00000080:00200000:2.0:1789760795.598403:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.598420:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6892, 6892] 00020000:00200000:2.0:1789760795.598428:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28229632 chunk: [28229632, 28233728) 28233728, 4096 00020000:00200000:2.0:1789760795.598433:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.598452:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6892, 6892] 00000008:00010000:2.0:1789760795.598471:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.598478:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.598490:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28229632, 28233728) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.598502:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28229632=0x1aec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.599141:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.599147:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28233728 00000020:00010000:2.0:1789760795.599157:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.599161:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.599164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.599193:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.599197:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.599215:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.599223:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.599257:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760795.599392:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266d80 - object 0x240000400:4 offset 28229632<>28233728 00000100:00100000:3.0:1789760795.599747:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.599771:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495966720:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760795.636296:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760795.636311:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.636323:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.636348:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760795.636381:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495966720:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.636552:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6892, 6892] 00000080:00200000:2.0:1789760795.636570:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.636577:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.636594:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28233728 i_blocks 55305 00000020:00200000:2.0:1789760795.636599:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.636609:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.636645:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28233728, count: 4096 00000080:00200000:2.0:1789760795.636654:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28233728, bytes: 4096 00000080:00200000:2.0:1789760795.636666:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.636686:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.636689:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28233728, 28237824) 0 0 00000080:00200000:2.0:1789760795.636694:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.636719:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6893, 6893] 00020000:00200000:2.0:1789760795.636727:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28233728 chunk: [28233728, 28237824) 28237824, 4096 00020000:00200000:2.0:1789760795.636732:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.636750:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6893, 6893] 00000008:00010000:2.0:1789760795.636769:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.636775:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.636787:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28233728, 28237824) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.636802:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28233728=0x1aed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.636856:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.636859:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28237824 00000020:00010000:2.0:1789760795.636867:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.636871:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.636874:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.636877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.636880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.636887:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.636894:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.636910:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760795.637034:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264700 - object 0x240000400:4 offset 28233728<>28237824 00000100:00100000:3.0:1789760795.637448:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495966848:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760795.672127:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974210 gen 2 00000008:00010000:3.0:1789760795.672228:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760795.672243:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.672252:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.672274:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760795.672305:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495966848:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.673874:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6893, 6893] 00000080:00200000:2.0:1789760795.673903:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.673911:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.673933:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28237824 i_blocks 55305 00000020:00200000:2.0:1789760795.673938:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.673956:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.673999:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28237824, count: 4096 00000080:00200000:2.0:1789760795.674008:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28237824, bytes: 4096 00000080:00200000:2.0:1789760795.674023:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.674032:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.674034:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28237824, 28241920) 0 0 00000080:00200000:2.0:1789760795.674039:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.674058:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6894, 6894] 00020000:00200000:2.0:1789760795.674067:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28237824 chunk: [28237824, 28241920) 28241920, 4096 00020000:00200000:2.0:1789760795.674071:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.674097:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6894, 6894] 00000008:00010000:2.0:1789760795.674118:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.674124:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.674136:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28237824, 28241920) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.674153:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28237824=0x1aee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.674749:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.674753:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28241920 00000020:00010000:2.0:1789760795.674764:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.674767:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.674771:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.674775:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.674778:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.674787:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.674794:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.674819:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760795.674949:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266300 - object 0x240000400:4 offset 28237824<>28241920 00000100:00100000:3.0:1789760795.675380:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.675408:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495966976:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760795.702919:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974211 gen 2 00000008:00010000:3.0:1789760795.702972:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760795.702981:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.702992:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.703017:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760795.703052:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495966976:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760795.703193:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6894, 6894] 00000080:00200000:2.0:1789760795.703220:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.703227:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.703312:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28241920 i_blocks 55305 00000020:00200000:2.0:1789760795.703318:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.703328:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.703363:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28241920, count: 4096 00000080:00200000:2.0:1789760795.703370:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28241920, bytes: 4096 00000080:00200000:2.0:1789760795.703379:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.703388:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.703390:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28241920, 28246016) 0 0 00000080:00200000:2.0:1789760795.703394:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.703410:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6895, 6895] 00020000:00200000:2.0:1789760795.703419:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28241920 chunk: [28241920, 28246016) 28246016, 4096 00020000:00200000:2.0:1789760795.703423:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.703443:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6895, 6895] 00000008:00010000:2.0:1789760795.703462:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.703467:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760795.703480:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28241920, 28246016) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.703496:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28241920=0x1aef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.703555:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.703558:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28246016 00000020:00010000:2.0:1789760795.703566:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.703570:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.703574:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.703577:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.703580:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.703588:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.703595:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.703612:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760795.705156:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264000 - object 0x240000400:4 offset 28241920<>28246016 00000100:00100000:3.0:1789760795.705862:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495967104:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760795.766513:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974212 gen 2 00000008:00010000:3.0:1789760795.766583:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760795.766593:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.766784:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.766817:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760795.766962:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495967104:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.767085:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6895, 6895] 00000080:00200000:1.0:1789760795.767102:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.767109:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.767126:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28246016 i_blocks 55305 00000020:00200000:1.0:1789760795.767130:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.767140:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.767214:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28246016, count: 4096 00000080:00200000:1.0:1789760795.767221:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28246016, bytes: 4096 00000080:00200000:1.0:1789760795.767401:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.767411:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.767414:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28246016, 28250112) 0 0 00000080:00200000:1.0:1789760795.767419:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.767436:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6896, 6896] 00020000:00200000:1.0:1789760795.767447:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28246016 chunk: [28246016, 28250112) 28250112, 4096 00020000:00200000:1.0:1789760795.767452:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.767476:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6896, 6896] 00000008:00010000:1.0:1789760795.767496:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.767501:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.767513:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28246016, 28250112) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.768056:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28246016=0x1af0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.768146:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.768149:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28250112 00000020:00010000:1.0:1789760795.768160:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.768164:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.768348:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.768354:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.768356:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.768368:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.768376:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.768593:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760795.771337:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 28246016<>28250112 00000100:00100000:0.0:1789760795.771624:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967232:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760795.816748:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760795.816765:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.816777:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.816797:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760795.816827:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967232:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.816923:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6896, 6896] 00000080:00200000:1.0:1789760795.816939:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.816945:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.816958:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28250112 i_blocks 55305 00000020:00200000:1.0:1789760795.816962:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.816970:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.817004:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28250112, count: 4096 00000080:00200000:1.0:1789760795.817011:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28250112, bytes: 4096 00000080:00200000:1.0:1789760795.817021:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.817030:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.817032:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28250112, 28254208) 0 0 00000080:00200000:1.0:1789760795.817036:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.817055:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6897, 6897] 00020000:00200000:1.0:1789760795.817064:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28250112 chunk: [28250112, 28254208) 28254208, 4096 00020000:00200000:1.0:1789760795.817068:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.817088:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6897, 6897] 00000008:00010000:1.0:1789760795.817106:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.817112:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.817123:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28250112, 28254208) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.817137:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28250112=0x1af1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.817292:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.817295:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28254208 00000020:00010000:1.0:1789760795.817305:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.817309:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.817312:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.817315:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.817318:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.817325:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.817333:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.817350:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760795.817484:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 28250112<>28254208 00000100:00100000:0.0:1789760795.817643:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760795.817661:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967360:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.859254:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974213 gen 2 00000100:00100000:0.0:1789760795.859295:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d880 x1876699495967360/t4294974214(4294974214) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760811 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760795.859343:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760795.859350:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.859360:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.859382:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760795.859406:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967360:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.859555:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6897, 6897] 00000080:00200000:1.0:1789760795.859572:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.859578:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.859594:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28254208 i_blocks 55305 00000020:00200000:1.0:1789760795.859597:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.859607:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.859640:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28254208, count: 4096 00000080:00200000:1.0:1789760795.859647:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28254208, bytes: 4096 00000080:00200000:1.0:1789760795.859657:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.859670:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.859673:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28254208, 28258304) 0 0 00000080:00200000:1.0:1789760795.859677:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.859694:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6898, 6898] 00020000:00200000:1.0:1789760795.859703:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28254208 chunk: [28254208, 28258304) 28258304, 4096 00020000:00200000:1.0:1789760795.859708:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.859729:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6898, 6898] 00000008:00010000:1.0:1789760795.859746:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.859751:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.859763:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28254208, 28258304) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.859777:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28254208=0x1af2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.859845:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.859855:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28258304 00000020:00010000:1.0:1789760795.859864:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.859867:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.859870:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.859873:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.859876:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.859884:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.859891:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.859910:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760795.860038:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 28254208<>28258304 00000100:00100000:0.0:1789760795.861010:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967488:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.897781:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974214 gen 2 00000100:00100000:0.0:1789760795.897821:0:2357:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504c000 x1876699495967488/t4294974215(4294974215) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760811 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760795.897878:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760795.897887:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.897896:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.897916:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760795.897941:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967488:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.898098:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6898, 6898] 00000080:00200000:1.0:1789760795.898115:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.898121:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.898138:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28258304 i_blocks 55305 00000020:00200000:1.0:1789760795.898141:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.898150:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.898223:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28258304, count: 4096 00000080:00200000:1.0:1789760795.898231:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28258304, bytes: 4096 00000080:00200000:1.0:1789760795.898241:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.898250:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.898252:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28258304, 28262400) 0 0 00000080:00200000:1.0:1789760795.898256:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.898274:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6899, 6899] 00020000:00200000:1.0:1789760795.898283:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28258304 chunk: [28258304, 28262400) 28262400, 4096 00020000:00200000:1.0:1789760795.898287:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.898307:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6899, 6899] 00000008:00010000:1.0:1789760795.898325:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.898331:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.898344:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28258304, 28262400) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.898357:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28258304=0x1af3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.898417:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.898420:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28262400 00000020:00010000:1.0:1789760795.898429:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.898432:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.898435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.898439:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.898442:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.898450:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.898458:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.898475:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760795.898605:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 28258304<>28262400 00000100:00100000:0.0:1789760795.899151:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760795.899223:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967616:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.936507:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974216 gen 2 00000008:00010000:0.0:1789760795.937135:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760795.937145:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.937157:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.937419:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6899, 6899] 00000008:00010000:0.0:1789760795.937422:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000080:00200000:1.0:1789760795.937438:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.937444:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:0.0:1789760795.937451:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967616:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.937459:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28262400 i_blocks 55305 00000020:00200000:1.0:1789760795.937462:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.937471:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.937504:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28262400, count: 4096 00000080:00200000:1.0:1789760795.937511:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28262400, bytes: 4096 00000080:00200000:1.0:1789760795.937521:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.937530:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.937532:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28262400, 28266496) 0 0 00000080:00200000:1.0:1789760795.937536:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.937552:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6900, 6900] 00020000:00200000:1.0:1789760795.937560:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28262400 chunk: [28262400, 28266496) 28266496, 4096 00020000:00200000:1.0:1789760795.937566:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.937587:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6900, 6900] 00000008:00010000:1.0:1789760795.937606:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.937612:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.937623:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28262400, 28266496) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.937637:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28262400=0x1af4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.937692:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.937695:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28266496 00000020:00010000:1.0:1789760795.937703:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.937707:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.937710:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.937713:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.937715:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.937722:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.937729:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.937753:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760795.938099:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 28262400<>28266496 00000100:00100000:0.0:1789760795.938263:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967744:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760795.987407:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974217 gen 2 00000008:00010000:0.0:1789760795.987464:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760795.987472:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.987484:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.987547:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760795.987576:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495967744:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760795.987661:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6900, 6900] 00000080:00200000:1.0:1789760795.987677:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760795.987684:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760795.987698:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28266496 i_blocks 55305 00000020:00200000:1.0:1789760795.987702:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760795.987711:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760795.988038:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28266496, count: 4096 00000080:00200000:1.0:1789760795.988048:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28266496, bytes: 4096 00000080:00200000:1.0:1789760795.988062:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760795.988072:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760795.988074:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28266496, 28270592) 0 0 00000080:00200000:1.0:1789760795.988079:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760795.988097:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6901, 6901] 00020000:00200000:1.0:1789760795.988106:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28266496 chunk: [28266496, 28270592) 28270592, 4096 00020000:00200000:1.0:1789760795.988110:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760795.988131:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6901, 6901] 00000008:00010000:1.0:1789760795.988150:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760795.988156:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760795.988166:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28266496, 28270592) append 0 append lockpos 0 00000080:00200000:1.0:1789760795.988214:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28266496=0x1af5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760795.988271:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760795.988274:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28270592 00000020:00010000:1.0:1789760795.988282:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760795.988286:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.988289:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760795.988292:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760795.988295:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760795.988377:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760795.988384:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760795.988399:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760795.988525:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 28266496<>28270592 00000100:00100000:0.0:1789760795.988593:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495967872:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760796.031954:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974218 gen 2 00000008:00010000:0.0:1789760796.032006:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760796.032014:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.032025:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.032090:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760796.032120:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495967872:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760796.032383:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6901, 6901] 00000080:00200000:1.0:1789760796.032400:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.032407:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.032423:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28270592 i_blocks 55305 00000020:00200000:1.0:1789760796.032426:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.032435:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.032470:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28270592, count: 4096 00000080:00200000:1.0:1789760796.032477:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28270592, bytes: 4096 00000080:00200000:1.0:1789760796.032489:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.032499:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760796.032501:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28270592, 28274688) 0 0 00000080:00200000:1.0:1789760796.032505:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.032584:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6902, 6902] 00020000:00200000:1.0:1789760796.032594:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28270592 chunk: [28270592, 28274688) 28274688, 4096 00020000:00200000:1.0:1789760796.032599:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.032620:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6902, 6902] 00000008:00010000:1.0:1789760796.032638:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.032644:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760796.032655:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28270592, 28274688) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.032671:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28270592=0x1af6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.032725:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.032728:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28274688 00000020:00010000:1.0:1789760796.032737:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.032741:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.032744:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.032747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.032750:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.032757:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.032764:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.032780:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:1.0:1789760796.032972:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 28270592<>28274688 00000100:00100000:0.0:1789760796.033043:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495968000:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760796.061545:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974219 gen 2 00000008:00010000:0.0:1789760796.061598:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760796.061606:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.061616:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.061692:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760796.061728:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495968000:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760796.061918:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6902, 6902] 00000080:00200000:2.0:1789760796.061935:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.061942:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.061958:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28274688 i_blocks 55305 00000020:00200000:2.0:1789760796.061962:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.061972:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.062008:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28274688, count: 4096 00000080:00200000:2.0:1789760796.062015:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28274688, bytes: 4096 00000080:00200000:2.0:1789760796.062025:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.062034:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760796.062037:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28274688, 28278784) 0 0 00000080:00200000:2.0:1789760796.062041:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.062057:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6903, 6903] 00020000:00200000:2.0:1789760796.062065:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28274688 chunk: [28274688, 28278784) 28278784, 4096 00020000:00200000:2.0:1789760796.062070:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.062089:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6903, 6903] 00000008:00010000:2.0:1789760796.062108:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.062113:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760796.062123:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28274688, 28278784) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.062137:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28274688=0x1af7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.062361:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.062364:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28278784 00000020:00010000:2.0:1789760796.062373:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.062377:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.062380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.062384:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.062387:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.062394:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.062401:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.062418:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760796.062606:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 28274688<>28278784 00000100:00100000:3.0:1789760796.062661:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495968128:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760796.091393:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974220 gen 2 00000008:00010000:2.0:1789760796.091524:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:2.0:1789760796.091534:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.091546:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.091569:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:2.0:1789760796.091599:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495968128:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760796.091606:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6903, 6903] 00000080:00200000:0.0:1789760796.091623:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.091629:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.091645:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28278784 i_blocks 55305 00000020:00200000:0.0:1789760796.091649:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.091659:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.091699:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28278784, count: 4096 00000080:00200000:0.0:1789760796.091706:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28278784, bytes: 4096 00000080:00200000:0.0:1789760796.091717:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.091726:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.091728:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28278784, 28282880) 0 0 00000080:00200000:0.0:1789760796.091734:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.091747:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6904, 6904] 00020000:00200000:0.0:1789760796.091754:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28278784 chunk: [28278784, 28282880) 28282880, 4096 00020000:00200000:0.0:1789760796.091758:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.091774:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6904, 6904] 00000008:00010000:0.0:1789760796.091788:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.091792:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760796.091801:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28278784, 28282880) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.091813:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28278784=0x1af8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.091869:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.091872:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28282880 00000020:00010000:0.0:1789760796.091880:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.091883:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.091886:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.091889:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.091892:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.091900:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.091906:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.091922:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760796.092044:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 28278784<>28282880 00000100:00100000:0.0:1789760796.092088:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760796.092103:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495968256:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760796.107631:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760796.140143:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974221 gen 2 00000008:00010000:0.0:1789760796.140753:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760796.140761:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.140771:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.140800:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760796.140835:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495968256:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760796.141100:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6904, 6904] 00000080:00200000:2.0:1789760796.141119:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.141125:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.141142:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28282880 i_blocks 55305 00000020:00200000:2.0:1789760796.141145:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.141330:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.141521:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28282880, count: 4096 00000080:00200000:2.0:1789760796.141530:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28282880, bytes: 4096 00000080:00200000:2.0:1789760796.141548:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.141558:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760796.141560:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28282880, 28286976) 0 0 00000080:00200000:2.0:1789760796.141565:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.141584:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6905, 6905] 00020000:00200000:2.0:1789760796.141592:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28282880 chunk: [28282880, 28286976) 28286976, 4096 00020000:00200000:2.0:1789760796.141596:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.141617:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6905, 6905] 00000008:00010000:2.0:1789760796.141636:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.141641:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760796.141653:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28282880, 28286976) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.141667:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28282880=0x1af9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.141727:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.141730:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28286976 00000020:00010000:2.0:1789760796.141739:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.141743:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.141747:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.141751:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.141754:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.141761:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.141768:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.141785:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2088136c0 00000008:00100000:2.0:1789760796.141977:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264380 - object 0x240000400:4 offset 28282880<>28286976 00000100:00100000:3.0:1789760796.142152:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495968384:192.168.203.109@tcp:4:dd.0 00000008:00010000:3.0:1789760796.198113:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2088136c0 00000020:00200000:3.0:1789760796.198128:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.198141:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.198166:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2088136c0 00000100:00100000:3.0:1789760796.198247:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495968384:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760796.198350:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6905, 6905] 00000080:00200000:2.0:1789760796.198367:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.198373:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.198389:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28286976 i_blocks 55305 00000020:00200000:2.0:1789760796.198393:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.198402:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.198436:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28286976, count: 4096 00000080:00200000:2.0:1789760796.198444:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28286976, bytes: 4096 00000080:00200000:2.0:1789760796.198455:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.198463:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760796.198465:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28286976, 28291072) 0 0 00000080:00200000:2.0:1789760796.198469:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.198486:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6906, 6906] 00020000:00200000:2.0:1789760796.198494:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28286976 chunk: [28286976, 28291072) 28291072, 4096 00020000:00200000:2.0:1789760796.198499:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.198518:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6906, 6906] 00000008:00010000:2.0:1789760796.198536:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.198542:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760796.198554:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28286976, 28291072) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.198568:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28286976=0x1afa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.198623:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.198625:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28291072 00000020:00010000:2.0:1789760796.198635:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.198639:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.198642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.198646:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.198650:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.198656:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.198663:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.199588:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813540 00000008:00100000:2.0:1789760796.199734:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205266a00 - object 0x240000400:4 offset 28286976<>28291072 00000100:00100000:3.0:1789760796.199808:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.199909:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495968512:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760796.248861:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974223 gen 2 00000008:00010000:3.0:1789760796.248925:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813540 00000020:00200000:3.0:1789760796.248934:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.248945:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.249025:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813540 00000100:00100000:3.0:1789760796.249065:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495968512:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760796.250635:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6906, 6906] 00000080:00200000:0.0:1789760796.250661:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.250668:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.250747:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28291072 i_blocks 55305 00000020:00200000:0.0:1789760796.250754:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.250768:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.250804:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28291072, count: 4096 00000080:00200000:0.0:1789760796.250813:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28291072, bytes: 4096 00000080:00200000:0.0:1789760796.250826:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.250836:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.250838:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28291072, 28295168) 0 0 00000080:00200000:0.0:1789760796.250842:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.250908:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6907, 6907] 00020000:00200000:0.0:1789760796.250918:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28291072 chunk: [28291072, 28295168) 28295168, 4096 00020000:00200000:0.0:1789760796.250923:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.250945:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6907, 6907] 00000008:00010000:0.0:1789760796.250966:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.250971:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760796.250984:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28291072, 28295168) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.250999:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28291072=0x1afb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.251132:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.251135:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28295168 00000020:00010000:0.0:1789760796.251145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.251149:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.251153:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.251156:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2088147b8 (ffff8dd205fe0488/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.251159:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0488/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.251387:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.251396:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.251423:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760796.251968:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211836d80 - object 0x240000400:4 offset 28291072<>28295168 00000100:00100000:1.0:1789760796.252078:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495968640:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760796.303643:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760796.303660:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.303672:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.303697:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760796.303727:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495968640:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.304066:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6907, 6907] 00000080:00200000:3.0:1789760796.304084:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.304090:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.304105:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28295168 i_blocks 55305 00000020:00200000:3.0:1789760796.304110:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.304120:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.304223:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28295168, count: 4096 00000080:00200000:3.0:1789760796.304479:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28295168, bytes: 4096 00000080:00200000:3.0:1789760796.304496:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.304506:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.304509:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28295168, 28299264) 0 0 00000080:00200000:3.0:1789760796.304514:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.305055:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6908, 6908] 00020000:00200000:3.0:1789760796.305066:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28295168 chunk: [28295168, 28299264) 28299264, 4096 00020000:00200000:3.0:1789760796.305071:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.305094:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6908, 6908] 00000008:00010000:3.0:1789760796.305113:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.305119:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.305131:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28295168, 28299264) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.305145:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28295168=0x1afc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.306837:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.306847:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28299264 00000020:00010000:3.0:1789760796.306864:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.306868:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.306872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.306877:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.306879:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.306893:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.306903:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.306937:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250900 00000008:00100000:3.0:1789760796.307081:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 28295168<>28299264 00000100:00100000:2.0:1789760796.307494:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495968768:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.365154:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974225 gen 2 00000008:00010000:2.0:1789760796.365279:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250900 00000020:00200000:2.0:1789760796.365288:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.365298:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.365447:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250900 00000100:00100000:2.0:1789760796.365480:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495968768:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.365583:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6908, 6908] 00000080:00200000:3.0:1789760796.365611:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.365618:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.365635:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28299264 i_blocks 55305 00000020:00200000:3.0:1789760796.365638:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.365648:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.365683:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28299264, count: 4096 00000080:00200000:3.0:1789760796.365690:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28299264, bytes: 4096 00000080:00200000:3.0:1789760796.365701:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.365711:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.365713:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28299264, 28303360) 0 0 00000080:00200000:3.0:1789760796.365718:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.365736:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6909, 6909] 00020000:00200000:3.0:1789760796.365744:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28299264 chunk: [28299264, 28303360) 28303360, 4096 00020000:00200000:3.0:1789760796.365749:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.365769:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6909, 6909] 00000008:00010000:3.0:1789760796.365786:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.365791:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.365803:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28299264, 28303360) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.365817:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28299264=0x1afd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.365868:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.365871:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28303360 00000020:00010000:3.0:1789760796.365878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.365881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.365884:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.365887:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.365890:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.365897:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.365903:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.365918:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250780 00000008:00100000:3.0:1789760796.366034:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 28299264<>28303360 00000100:00100000:2.0:1789760796.366223:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.366239:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495968896:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.424534:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974226 gen 2 00000008:00010000:2.0:1789760796.424584:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250780 00000020:00200000:2.0:1789760796.424603:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.424615:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.424635:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250780 00000100:00100000:2.0:1789760796.424663:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495968896:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.424827:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6909, 6909] 00000080:00200000:3.0:1789760796.424845:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.424851:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.424866:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28303360 i_blocks 55305 00000020:00200000:3.0:1789760796.424869:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.424880:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.424913:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28303360, count: 4096 00000080:00200000:3.0:1789760796.424920:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28303360, bytes: 4096 00000080:00200000:3.0:1789760796.424929:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.424938:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.424941:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28303360, 28307456) 0 0 00000080:00200000:3.0:1789760796.424946:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.424962:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6910, 6910] 00020000:00200000:3.0:1789760796.424971:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28303360 chunk: [28303360, 28307456) 28307456, 4096 00020000:00200000:3.0:1789760796.424976:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.424995:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6910, 6910] 00000008:00010000:3.0:1789760796.425014:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.425020:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.425031:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28303360, 28307456) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.425045:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28303360=0x1afe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.425100:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.425102:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28307456 00000020:00010000:3.0:1789760796.425111:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.425114:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.425118:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.425122:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.425124:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.425132:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.425140:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.425156:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250b40 00000008:00100000:3.0:1789760796.425344:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 28303360<>28307456 00000100:00100000:2.0:1789760796.425607:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969024:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.454511:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974227 gen 2 00000008:00010000:2.0:1789760796.454561:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250b40 00000020:00200000:2.0:1789760796.454570:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.454582:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.454678:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250b40 00000100:00100000:2.0:1789760796.454709:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969024:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.454864:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6910, 6910] 00000080:00200000:3.0:1789760796.454881:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.454887:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.454904:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28307456 i_blocks 55305 00000020:00200000:3.0:1789760796.454907:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.454917:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.454950:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28307456, count: 4096 00000080:00200000:3.0:1789760796.454958:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28307456, bytes: 4096 00000080:00200000:3.0:1789760796.454967:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.454976:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.454978:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28307456, 28311552) 0 0 00000080:00200000:3.0:1789760796.454983:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.455000:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6911, 6911] 00020000:00200000:3.0:1789760796.455009:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28307456 chunk: [28307456, 28311552) 28311552, 4096 00020000:00200000:3.0:1789760796.455014:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.455033:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6911, 6911] 00000008:00010000:3.0:1789760796.455050:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.455056:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.455068:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28307456, 28311552) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.455081:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28307456=0x1aff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.461342:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.461356:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28311552 00000020:00010000:3.0:1789760796.461375:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.461380:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.461386:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.461391:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.461394:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.461410:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.461422:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.461460:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250840 00000008:00100000:3.0:1789760796.461739:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 28307456<>28311552 00000100:00100000:2.0:1789760796.461948:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.461974:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969152:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.514422:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974228 gen 2 00000008:00010000:2.0:1789760796.514475:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250840 00000020:00200000:2.0:1789760796.514484:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.514496:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.514522:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250840 00000100:00100000:2.0:1789760796.514553:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969152:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.514684:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6911, 6911] 00000080:00200000:3.0:1789760796.514701:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.514707:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.514724:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28311552 i_blocks 55305 00000020:00200000:3.0:1789760796.514728:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.514737:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.514772:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28311552, count: 4096 00000080:00200000:3.0:1789760796.514779:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28311552, bytes: 4096 00000080:00200000:3.0:1789760796.514792:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.514801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.514804:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28311552, 28315648) 0 0 00000080:00200000:3.0:1789760796.514809:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.514828:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6912, 6912] 00020000:00200000:3.0:1789760796.514891:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28311552 chunk: [28311552, 28315648) 28315648, 4096 00020000:00200000:3.0:1789760796.514896:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.514917:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6912, 6912] 00000008:00010000:3.0:1789760796.514937:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.514943:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.514955:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28311552, 28315648) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.514969:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28311552=0x1b00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.515028:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.515031:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28315648 00000020:00010000:3.0:1789760796.515040:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.515043:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.515047:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.515051:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.515054:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.515061:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.515067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.515085:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250cc0 00000008:00100000:3.0:1789760796.515263:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 28311552<>28315648 00000100:00100000:2.0:1789760796.515322:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969280:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.555380:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974229 gen 2 00000008:00010000:2.0:1789760796.555434:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250cc0 00000020:00200000:2.0:1789760796.555444:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.555458:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.555485:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250cc0 00000100:00100000:2.0:1789760796.555514:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969280:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760796.555525:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6912, 6912] 00000080:00200000:0.0:1789760796.555547:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.555554:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.555572:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28315648 i_blocks 55305 00000020:00200000:0.0:1789760796.555576:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.555587:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.555624:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28315648, count: 4096 00000080:00200000:0.0:1789760796.555630:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28315648, bytes: 4096 00000080:00200000:0.0:1789760796.555640:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.555659:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.555661:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28315648, 28319744) 0 0 00000080:00200000:0.0:1789760796.555665:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.555682:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6913, 6913] 00020000:00200000:0.0:1789760796.555689:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28315648 chunk: [28315648, 28319744) 28319744, 4096 00020000:00200000:0.0:1789760796.555693:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.555711:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6913, 6913] 00000008:00010000:0.0:1789760796.555728:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.555733:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760796.555746:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28315648, 28319744) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.555760:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28315648=0x1b01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.555815:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.555817:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28319744 00000020:00010000:0.0:1789760796.555825:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.555828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.555832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.555835:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.555838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.555845:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.555851:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.555866:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760796.555981:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835f80 - object 0x240000400:4 offset 28315648<>28319744 00000100:00100000:0.0:1789760796.556026:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760796.556040:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495969408:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.556153:0:468:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1789760796.556163:0:468:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.556211:0:468:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:0.0:1789760796.590398:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760796.590415:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.590428:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.590456:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760796.590490:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495969408:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.590499:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6913, 6913] 00000080:00200000:3.0:1789760796.590581:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.590588:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.590606:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28319744 i_blocks 57353 00000020:00200000:3.0:1789760796.590610:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.590621:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.590656:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28319744, count: 4096 00000080:00200000:3.0:1789760796.590664:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28319744, bytes: 4096 00000080:00200000:3.0:1789760796.590749:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.590760:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.590762:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28319744, 28323840) 0 0 00000080:00200000:3.0:1789760796.590766:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.590783:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6914, 6914] 00020000:00200000:3.0:1789760796.590791:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28319744 chunk: [28319744, 28323840) 28323840, 4096 00020000:00200000:3.0:1789760796.590796:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.590816:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6914, 6914] 00000008:00010000:3.0:1789760796.590837:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.590842:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.590910:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28319744, 28323840) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.590923:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28319744=0x1b02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.591067:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.591070:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28323840 00000020:00010000:3.0:1789760796.591079:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.591082:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.591086:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.591089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.591092:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.591099:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.591105:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.591121:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250d80 00000008:00100000:3.0:1789760796.591297:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203145880 - object 0x240000400:4 offset 28319744<>28323840 00000100:00100000:2.0:1789760796.591376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495969536:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.662403:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974230 gen 2 00000100:00100000:2.0:1789760796.662450:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203145880 x1876699495969536/t4294974231(4294974231) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760796.662500:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250d80 00000020:00200000:2.0:1789760796.662508:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.662530:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.662550:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250d80 00000100:00100000:2.0:1789760796.662577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495969536:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.662679:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6914, 6914] 00000080:00200000:3.0:1789760796.662697:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.662703:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.662718:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28323840 i_blocks 57353 00000020:00200000:3.0:1789760796.662722:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.662732:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.662768:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28323840, count: 4096 00000080:00200000:3.0:1789760796.662776:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28323840, bytes: 4096 00000080:00200000:3.0:1789760796.662788:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.662799:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.662801:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28323840, 28327936) 0 0 00000080:00200000:3.0:1789760796.662806:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.662825:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6915, 6915] 00020000:00200000:3.0:1789760796.662834:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28323840 chunk: [28323840, 28327936) 28327936, 4096 00020000:00200000:3.0:1789760796.662839:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.662861:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6915, 6915] 00000008:00010000:3.0:1789760796.662882:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.662887:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.662898:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28323840, 28327936) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.662913:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28323840=0x1b03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.662987:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.662990:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28327936 00000020:00010000:3.0:1789760796.662998:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.663001:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.663005:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.663008:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.663011:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.663018:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.663025:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.663042:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250600 00000008:00100000:3.0:1789760796.663353:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147800 - object 0x240000400:4 offset 28323840<>28327936 00000100:00100000:2.0:1789760796.663781:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969664:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.708702:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974231 gen 2 00000100:00100000:2.0:1789760796.708738:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203147800 x1876699495969664/t4294974232(4294974232) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760796.708784:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250600 00000020:00200000:2.0:1789760796.708791:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.708802:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.708875:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250600 00000100:00100000:2.0:1789760796.708900:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969664:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.709479:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6915, 6915] 00000080:00200000:3.0:1789760796.709496:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.709503:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.709527:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28327936 i_blocks 57353 00000020:00200000:3.0:1789760796.709530:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.709540:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.709573:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28327936, count: 4096 00000080:00200000:3.0:1789760796.709581:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28327936, bytes: 4096 00000080:00200000:3.0:1789760796.709592:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.709600:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.709602:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28327936, 28332032) 0 0 00000080:00200000:3.0:1789760796.709606:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.709622:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6916, 6916] 00020000:00200000:3.0:1789760796.709629:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28327936 chunk: [28327936, 28332032) 28332032, 4096 00020000:00200000:3.0:1789760796.709634:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.709653:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6916, 6916] 00000008:00010000:3.0:1789760796.709671:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.709677:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.709689:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28327936, 28332032) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.709703:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28327936=0x1b04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.709765:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.709768:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28332032 00000020:00010000:3.0:1789760796.709777:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.709780:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.709783:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.709788:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.709791:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.709799:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.709807:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.709825:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250180 00000008:00100000:3.0:1789760796.710130:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147100 - object 0x240000400:4 offset 28327936<>28332032 00000100:00100000:2.0:1789760796.710438:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.710462:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969792:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.743732:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974233 gen 2 00000008:00010000:2.0:1789760796.743800:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250180 00000020:00200000:2.0:1789760796.743809:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.743819:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.743836:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250180 00000100:00100000:2.0:1789760796.743866:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969792:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.744362:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6916, 6916] 00000080:00200000:3.0:1789760796.744381:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.744387:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.744404:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28332032 i_blocks 57353 00000020:00200000:3.0:1789760796.744408:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.744418:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.744455:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28332032, count: 4096 00000080:00200000:3.0:1789760796.744462:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28332032, bytes: 4096 00000080:00200000:3.0:1789760796.744475:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.744485:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.744488:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28332032, 28336128) 0 0 00000080:00200000:3.0:1789760796.744493:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.744510:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6917, 6917] 00020000:00200000:3.0:1789760796.744529:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28332032 chunk: [28332032, 28336128) 28336128, 4096 00020000:00200000:3.0:1789760796.744535:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.744555:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6917, 6917] 00000008:00010000:3.0:1789760796.744574:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.744579:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.744592:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28332032, 28336128) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.744606:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28332032=0x1b05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.744668:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.744671:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28336128 00000020:00010000:3.0:1789760796.744679:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.744682:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.744686:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.744690:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.744693:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.744701:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.744708:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.744725:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250a80 00000008:00100000:3.0:1789760796.744871:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203144000 - object 0x240000400:4 offset 28332032<>28336128 00000100:00100000:2.0:1789760796.745110:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969920:192.168.203.109@tcp:4:dd.0 00000008:00010000:2.0:1789760796.782910:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250a80 00000020:00200000:2.0:1789760796.782927:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.782939:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.783075:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250a80 00000100:00100000:2.0:1789760796.783109:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495969920:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760796.783665:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6917, 6917] 00000080:00200000:3.0:1789760796.783684:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.783690:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.783704:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28336128 i_blocks 57353 00000020:00200000:3.0:1789760796.783708:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.783717:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.783752:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28336128, count: 4096 00000080:00200000:3.0:1789760796.783759:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28336128, bytes: 4096 00000080:00200000:3.0:1789760796.783771:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.783780:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.783782:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28336128, 28340224) 0 0 00000080:00200000:3.0:1789760796.783787:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.783805:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6918, 6918] 00020000:00200000:3.0:1789760796.783814:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28336128 chunk: [28336128, 28340224) 28340224, 4096 00020000:00200000:3.0:1789760796.783819:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.783841:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6918, 6918] 00000008:00010000:3.0:1789760796.784117:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.784123:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760796.784136:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28336128, 28340224) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.784152:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28336128=0x1b06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.784318:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.784322:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28340224 00000020:00010000:3.0:1789760796.784331:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.784334:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.784337:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.784341:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.784344:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.784353:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.784360:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.784384:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208250480 00000008:00100000:3.0:1789760796.784594:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203147480 - object 0x240000400:4 offset 28336128<>28340224 00000100:00100000:2.0:1789760796.784676:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495970048:192.168.203.109@tcp:4:dd.0 00000100:00100000:2.0:1789760796.818648:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974234 gen 2 00000100:00100000:2.0:1789760796.818688:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd203147480 x1876699495970048/t4294974235(4294974235) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760796.818737:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208250480 00000020:00200000:2.0:1789760796.818745:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.818755:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.818771:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208250480 00000100:00100000:2.0:1789760796.818795:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495970048:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760796.818876:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6918, 6918] 00000080:00200000:1.0:1789760796.818892:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.818898:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.818913:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28340224 i_blocks 57353 00000020:00200000:1.0:1789760796.818916:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.818925:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.819026:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28340224, count: 4096 00000080:00200000:1.0:1789760796.819034:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28340224, bytes: 4096 00000080:00200000:1.0:1789760796.819045:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.819053:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760796.819055:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28340224, 28344320) 0 0 00000080:00200000:1.0:1789760796.819060:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.819078:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6919, 6919] 00020000:00200000:1.0:1789760796.819086:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28340224 chunk: [28340224, 28344320) 28344320, 4096 00020000:00200000:1.0:1789760796.819091:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.819110:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6919, 6919] 00000008:00010000:1.0:1789760796.819127:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.819133:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760796.819144:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28340224, 28344320) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.819160:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28340224=0x1b07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.819317:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.819320:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28344320 00000020:00010000:1.0:1789760796.819329:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.819332:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.819336:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.819339:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.819343:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.819350:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.819357:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.819373:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760796.819500:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ca80 - object 0x240000400:4 offset 28340224<>28344320 00000100:00100000:0.0:1789760796.819799:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495970176:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760796.871834:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974236 gen 2 00000008:00010000:0.0:1789760796.871912:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760796.871921:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.871931:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.872139:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6919, 6919] 00000080:00200000:1.0:1789760796.872159:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.872166:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.872222:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28344320 i_blocks 57353 00000020:00200000:1.0:1789760796.872225:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.872236:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.872272:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28344320, count: 4096 00000080:00200000:1.0:1789760796.872281:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28344320, bytes: 4096 00000080:00200000:1.0:1789760796.872293:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.872302:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760796.872305:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28344320, 28348416) 0 0 00000080:00200000:1.0:1789760796.872310:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.872342:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6920, 6920] 00020000:00200000:1.0:1789760796.872352:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28344320 chunk: [28344320, 28348416) 28348416, 4096 00020000:00200000:1.0:1789760796.872357:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.872378:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6920, 6920] 00000008:00010000:1.0:1789760796.872396:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.872402:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760796.872413:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28344320, 28348416) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.872428:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28344320=0x1b08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.872489:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.872492:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28348416 00000020:00010000:1.0:1789760796.872501:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.872505:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.872508:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.872511:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.872642:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.872653:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.872659:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.872681:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760796.872881:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504dc00 - object 0x240000400:4 offset 28344320<>28348416 00000100:00100000:1.0:1789760796.872932:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495970304:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760796.874147:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760796.874216:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495970176:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760796.921866:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974237 gen 2 00000008:00010000:0.0:1789760796.921921:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760796.921930:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.921943:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.922402:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6920, 6920] 00000080:00200000:1.0:1789760796.922422:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.922429:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.922446:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28348416 i_blocks 57353 00000020:00200000:1.0:1789760796.922450:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.922460:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.922494:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28348416, count: 4096 00000080:00200000:1.0:1789760796.922503:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28348416, bytes: 4096 00000008:00010000:0.0:1789760796.922592:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760796.922628:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495970304:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760796.922711:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.922722:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760796.922724:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28348416, 28352512) 0 0 00000080:00200000:1.0:1789760796.922729:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.922747:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6921, 6921] 00020000:00200000:1.0:1789760796.922755:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28348416 chunk: [28348416, 28352512) 28352512, 4096 00020000:00200000:1.0:1789760796.922760:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.922781:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6921, 6921] 00000008:00010000:1.0:1789760796.922801:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.922805:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760796.922816:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28348416, 28352512) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.922827:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28348416=0x1b09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.922881:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.922884:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28352512 00000020:00010000:1.0:1789760796.922894:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.922897:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.922900:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.922904:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.923443:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.923452:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.923459:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.923477:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760796.923611:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c380 - object 0x240000400:4 offset 28348416<>28352512 00000100:00100000:0.0:1789760796.923791:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495970432:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760796.963597:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974238 gen 2 00000008:00010000:0.0:1789760796.963650:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760796.963660:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.963671:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.963701:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760796.963731:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495970432:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760796.964215:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6921, 6921] 00000080:00200000:1.0:1789760796.964234:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.964240:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.964257:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28352512 i_blocks 57353 00000020:00200000:1.0:1789760796.964262:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.964272:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.964307:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28352512, count: 4096 00000080:00200000:1.0:1789760796.964314:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28352512, bytes: 4096 00000080:00200000:1.0:1789760796.964630:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.964640:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760796.964642:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28352512, 28356608) 0 0 00000080:00200000:1.0:1789760796.964647:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.964663:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6922, 6922] 00020000:00200000:1.0:1789760796.964829:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28352512 chunk: [28352512, 28356608) 28356608, 4096 00020000:00200000:1.0:1789760796.964835:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.964866:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6922, 6922] 00000008:00010000:1.0:1789760796.964939:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.964945:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760796.967679:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28352512, 28356608) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.967708:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28352512=0x1b0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.967792:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.967796:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28356608 00000020:00010000:1.0:1789760796.967805:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.967809:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.967812:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.967816:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.967819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.967828:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.967838:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.967880:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760796.968030:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ea00 - object 0x240000400:4 offset 28352512<>28356608 00000100:00100000:0.0:1789760796.969640:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495970560:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.039577:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974239 gen 2 00000008:00010000:0.0:1789760797.039629:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760797.039637:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.039649:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.039668:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760797.039695:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495970560:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.039789:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6922, 6922] 00000080:00200000:1.0:1789760797.039805:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.039811:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.039829:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28356608 i_blocks 57353 00000020:00200000:1.0:1789760797.039832:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.039841:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.039935:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28356608, count: 4096 00000080:00200000:1.0:1789760797.039943:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28356608, bytes: 4096 00000080:00200000:1.0:1789760797.039953:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.039962:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.039965:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28356608, 28360704) 0 0 00000080:00200000:1.0:1789760797.039969:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.039985:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6923, 6923] 00020000:00200000:1.0:1789760797.039993:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28356608 chunk: [28356608, 28360704) 28360704, 4096 00020000:00200000:1.0:1789760797.039999:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.040019:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6923, 6923] 00000008:00010000:1.0:1789760797.040038:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.040043:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.040055:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28356608, 28360704) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.040073:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28356608=0x1b0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.040130:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.040133:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28360704 00000020:00010000:1.0:1789760797.040141:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.040145:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.040148:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.040152:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.040155:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.040162:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.040319:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.040336:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760797.040461:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c000 - object 0x240000400:4 offset 28356608<>28360704 00000100:00100000:0.0:1789760797.040622:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495970688:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.076238:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974240 gen 2 00000008:00010000:0.0:1789760797.076291:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760797.076299:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.076310:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.076333:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760797.076364:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495970688:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.076522:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6923, 6923] 00000080:00200000:1.0:1789760797.076543:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.076550:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.076566:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28360704 i_blocks 57353 00000020:00200000:1.0:1789760797.076570:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.076579:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.076612:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28360704, count: 4096 00000080:00200000:1.0:1789760797.076620:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28360704, bytes: 4096 00000080:00200000:1.0:1789760797.076629:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.076639:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.076642:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28360704, 28364800) 0 0 00000080:00200000:1.0:1789760797.076646:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.076664:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6924, 6924] 00020000:00200000:1.0:1789760797.076673:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28360704 chunk: [28360704, 28364800) 28364800, 4096 00020000:00200000:1.0:1789760797.076679:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.076700:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6924, 6924] 00000008:00010000:1.0:1789760797.076718:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.076724:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.076736:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28360704, 28364800) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.076750:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28360704=0x1b0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.076806:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.076810:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28364800 00000020:00010000:1.0:1789760797.076819:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.076824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.076828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.076832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.076836:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.076844:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.076851:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.076869:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79780 00000008:00100000:1.0:1789760797.076999:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d880 - object 0x240000400:4 offset 28360704<>28364800 00000100:00100000:0.0:1789760797.077269:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495970816:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760797.118941:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79780 00000020:00200000:0.0:1789760797.118958:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.118971:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.118992:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79780 00000100:00100000:0.0:1789760797.119023:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495970816:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.119164:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6924, 6924] 00000080:00200000:1.0:1789760797.119229:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.119235:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.119250:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28364800 i_blocks 57353 00000020:00200000:1.0:1789760797.119254:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.119262:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.119297:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28364800, count: 4096 00000080:00200000:1.0:1789760797.119305:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28364800, bytes: 4096 00000080:00200000:1.0:1789760797.119316:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.119326:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.119329:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28364800, 28368896) 0 0 00000080:00200000:1.0:1789760797.119334:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.119352:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6925, 6925] 00020000:00200000:1.0:1789760797.119361:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28364800 chunk: [28364800, 28368896) 28368896, 4096 00020000:00200000:1.0:1789760797.119365:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.119387:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6925, 6925] 00000008:00010000:1.0:1789760797.119406:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.119412:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.119424:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28364800, 28368896) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.119439:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28364800=0x1b0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.119497:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.119500:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28368896 00000020:00010000:1.0:1789760797.119509:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.119749:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.119754:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.119758:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.119760:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.119770:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.119777:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.119799:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d799c0 00000008:00100000:1.0:1789760797.119939:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ed80 - object 0x240000400:4 offset 28364800<>28368896 00000100:00100000:0.0:1789760797.120011:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495970944:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760797.131786:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760797.158090:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974242 gen 2 00000008:00010000:0.0:1789760797.158159:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d799c0 00000020:00200000:0.0:1789760797.158218:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.158229:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.158251:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d799c0 00000100:00100000:0.0:1789760797.158280:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495970944:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.158423:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6925, 6925] 00000080:00200000:1.0:1789760797.158440:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.158446:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.158463:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28368896 i_blocks 57353 00000020:00200000:1.0:1789760797.158467:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.158476:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.158510:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28368896, count: 4096 00000080:00200000:1.0:1789760797.158527:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28368896, bytes: 4096 00000080:00200000:1.0:1789760797.158538:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.158548:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.158551:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28368896, 28372992) 0 0 00000080:00200000:1.0:1789760797.158555:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.158572:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6926, 6926] 00020000:00200000:1.0:1789760797.158580:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28368896 chunk: [28368896, 28372992) 28372992, 4096 00020000:00200000:1.0:1789760797.158584:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.158605:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6926, 6926] 00000008:00010000:1.0:1789760797.158623:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.158628:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.158639:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28368896, 28372992) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.158654:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28368896=0x1b0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.158710:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.158713:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28372992 00000020:00010000:1.0:1789760797.158722:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.158726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.158729:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.158733:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.158735:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.158742:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.158749:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.158766:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79540 00000008:00100000:1.0:1789760797.158892:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f100 - object 0x240000400:4 offset 28368896<>28372992 00000100:00100000:0.0:1789760797.159055:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971072:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760797.189936:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79540 00000020:00200000:0.0:1789760797.189954:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.189968:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.190014:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79540 00000100:00100000:0.0:1789760797.190049:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971072:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.190069:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6926, 6926] 00000080:00200000:1.0:1789760797.190086:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.190093:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.190108:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28372992 i_blocks 57353 00000020:00200000:1.0:1789760797.190112:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.190121:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.190156:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28372992, count: 4096 00000080:00200000:1.0:1789760797.190163:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28372992, bytes: 4096 00000080:00200000:1.0:1789760797.191482:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.191501:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.191504:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28372992, 28377088) 0 0 00000080:00200000:1.0:1789760797.191521:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.191544:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6927, 6927] 00020000:00200000:1.0:1789760797.191554:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28372992 chunk: [28372992, 28377088) 28377088, 4096 00020000:00200000:1.0:1789760797.191559:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.191583:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6927, 6927] 00000008:00010000:1.0:1789760797.191618:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.191624:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.191635:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28372992, 28377088) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.191651:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28372992=0x1b0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.191800:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.191804:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28377088 00000020:00010000:1.0:1789760797.191814:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.191818:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.191821:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.191824:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.191828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.191838:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.195743:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.195813:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79900 00000008:00100000:1.0:1789760797.195990:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504ce00 - object 0x240000400:4 offset 28372992<>28377088 00000100:00100000:0.0:1789760797.196274:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760797.196307:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971200:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.240728:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974244 gen 2 00000008:00010000:0.0:1789760797.240792:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79900 00000020:00200000:0.0:1789760797.240802:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.240813:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.240877:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79900 00000100:00100000:0.0:1789760797.240906:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971200:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.240980:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6927, 6927] 00000080:00200000:1.0:1789760797.240997:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.241003:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.241019:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28377088 i_blocks 57353 00000020:00200000:1.0:1789760797.241023:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.241032:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.241064:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28377088, count: 4096 00000080:00200000:1.0:1789760797.241072:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28377088, bytes: 4096 00000080:00200000:1.0:1789760797.241081:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.241091:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.241094:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28377088, 28381184) 0 0 00000080:00200000:1.0:1789760797.241098:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.241116:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6928, 6928] 00020000:00200000:1.0:1789760797.241676:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28377088 chunk: [28377088, 28381184) 28381184, 4096 00020000:00200000:1.0:1789760797.241683:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.241705:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6928, 6928] 00000008:00010000:1.0:1789760797.241724:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.241730:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.241742:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28377088, 28381184) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.241757:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28377088=0x1b10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.241817:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.241820:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28381184 00000020:00010000:1.0:1789760797.241828:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.241832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.241834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.241838:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.241842:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.241849:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.241857:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.241875:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760797.241997:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 28377088<>28381184 00000100:00100000:0.0:1789760797.242066:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971328:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.322747:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974245 gen 2 00000008:00010000:1.0:1789760797.323674:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:1.0:1789760797.323685:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.323697:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.323722:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:1.0:1789760797.323756:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971328:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760797.323866:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6928, 6928] 00000080:00200000:0.0:1789760797.323883:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.323891:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.323906:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28381184 i_blocks 57353 00000020:00200000:0.0:1789760797.323910:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.323927:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.323966:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28381184, count: 4096 00000080:00200000:0.0:1789760797.323975:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28381184, bytes: 4096 00000080:00200000:0.0:1789760797.323990:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.323998:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760797.324000:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28381184, 28385280) 0 0 00000080:00200000:0.0:1789760797.324005:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.324023:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6929, 6929] 00020000:00200000:0.0:1789760797.324031:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28381184 chunk: [28381184, 28385280) 28385280, 4096 00020000:00200000:0.0:1789760797.324036:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.324057:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6929, 6929] 00000008:00010000:0.0:1789760797.324076:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.324082:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760797.324093:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28381184, 28385280) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.324109:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28381184=0x1b11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.324218:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.324221:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28385280 00000020:00010000:0.0:1789760797.324241:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.324245:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.324249:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.324254:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.324257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.324264:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.324271:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.324379:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760797.324525:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 28381184<>28385280 00000100:00100000:1.0:1789760797.324942:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495971456:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760797.393809:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760797.393826:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.394788:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.394816:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760797.394852:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495971456:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.395159:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6929, 6929] 00000080:00200000:1.0:1789760797.395202:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.395209:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.395226:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28385280 i_blocks 57353 00000020:00200000:1.0:1789760797.395229:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.395240:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.395286:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28385280, count: 4096 00000080:00200000:1.0:1789760797.395487:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28385280, bytes: 4096 00000080:00200000:1.0:1789760797.395503:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.395717:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.395720:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28385280, 28389376) 0 0 00000080:00200000:1.0:1789760797.395725:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.395743:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6930, 6930] 00020000:00200000:1.0:1789760797.395751:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28385280 chunk: [28385280, 28389376) 28389376, 4096 00020000:00200000:1.0:1789760797.395756:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.395777:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6930, 6930] 00000008:00010000:1.0:1789760797.395797:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.395803:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.395814:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28385280, 28389376) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.395828:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28385280=0x1b12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.395894:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.395897:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28389376 00000020:00010000:1.0:1789760797.395906:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.395909:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.395912:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.395915:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.395918:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.395925:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.395931:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.395950:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79240 00000008:00100000:1.0:1789760797.396086:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504fb80 - object 0x240000400:4 offset 28385280<>28389376 00000100:00100000:0.0:1789760797.396213:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971584:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.436941:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974247 gen 2 00000008:00010000:0.0:1789760797.437013:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79240 00000020:00200000:0.0:1789760797.437022:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.437033:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.437051:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79240 00000100:00100000:0.0:1789760797.437078:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971584:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.437197:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6930, 6930] 00000080:00200000:1.0:1789760797.437214:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.437220:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.437237:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28389376 i_blocks 57353 00000020:00200000:1.0:1789760797.437241:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.437249:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.437282:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28389376, count: 4096 00000080:00200000:1.0:1789760797.437290:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28389376, bytes: 4096 00000080:00200000:1.0:1789760797.437300:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.437309:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.437311:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28389376, 28393472) 0 0 00000080:00200000:1.0:1789760797.437316:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.437332:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6931, 6931] 00020000:00200000:1.0:1789760797.437340:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28389376 chunk: [28389376, 28393472) 28393472, 4096 00020000:00200000:1.0:1789760797.437345:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.437366:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6931, 6931] 00000008:00010000:1.0:1789760797.437386:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.437392:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.437405:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28389376, 28393472) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.437421:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28389376=0x1b13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.437474:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.437477:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28393472 00000020:00010000:1.0:1789760797.437486:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.437490:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.437492:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.437496:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.437499:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.437506:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.437513:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.437602:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d793c0 00000008:00100000:0.0:1789760797.439404:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 28389376<>28393472 00000100:00100000:0.0:1789760797.439484:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760797.439508:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971712:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.493252:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974248 gen 2 00000008:00010000:0.0:1789760797.493300:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d793c0 00000020:00200000:0.0:1789760797.493308:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.493318:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.493337:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d793c0 00000100:00100000:0.0:1789760797.493364:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495971712:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.493461:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6931, 6931] 00000080:00200000:1.0:1789760797.493476:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.493482:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.493498:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28393472 i_blocks 57353 00000020:00200000:1.0:1789760797.493501:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.493510:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.493541:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28393472, count: 4096 00000080:00200000:1.0:1789760797.493547:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28393472, bytes: 4096 00000080:00200000:1.0:1789760797.493556:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.493564:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.493566:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28393472, 28397568) 0 0 00000080:00200000:1.0:1789760797.493571:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.493587:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6932, 6932] 00020000:00200000:1.0:1789760797.493594:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28393472 chunk: [28393472, 28397568) 28397568, 4096 00020000:00200000:1.0:1789760797.493598:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.493616:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6932, 6932] 00000008:00010000:1.0:1789760797.493632:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.493637:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.493647:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28393472, 28397568) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.493659:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28393472=0x1b14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.493707:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.493709:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28397568 00000020:00010000:1.0:1789760797.493716:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.493720:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.493723:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.493726:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.493729:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.493734:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.493740:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.493754:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79480 00000008:00100000:1.0:1789760797.493875:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504df80 - object 0x240000400:4 offset 28393472<>28397568 00000100:00100000:1.0:1789760797.493912:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760797.493926:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495971840:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.532642:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974249 gen 2 00000008:00010000:0.0:1789760797.532696:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79480 00000020:00200000:0.0:1789760797.532705:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.532715:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.532779:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79480 00000100:00100000:0.0:1789760797.532812:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495971840:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.533450:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6932, 6932] 00000080:00200000:1.0:1789760797.533469:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.533476:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.533493:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28397568 i_blocks 57353 00000020:00200000:1.0:1789760797.533496:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.533506:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.533610:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28397568, count: 4096 00000080:00200000:1.0:1789760797.533618:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28397568, bytes: 4096 00000080:00200000:1.0:1789760797.533630:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.533639:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.533641:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28397568, 28401664) 0 0 00000080:00200000:1.0:1789760797.533646:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.533663:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6933, 6933] 00020000:00200000:1.0:1789760797.533671:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28397568 chunk: [28397568, 28401664) 28401664, 4096 00020000:00200000:1.0:1789760797.533676:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.533697:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6933, 6933] 00000008:00010000:1.0:1789760797.533714:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.533719:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.533732:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28397568, 28401664) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.533747:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28397568=0x1b15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.533809:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.533813:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28401664 00000020:00010000:1.0:1789760797.533822:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.533826:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.533829:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.533832:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.533834:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.533842:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.533902:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.533925:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79a80 00000008:00100000:1.0:1789760797.534051:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504f800 - object 0x240000400:4 offset 28397568<>28401664 00000100:00100000:0.0:1789760797.534121:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495971968:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760797.569669:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79a80 00000020:00200000:0.0:1789760797.569685:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.569698:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.569768:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79a80 00000100:00100000:0.0:1789760797.569801:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495971968:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.571245:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6933, 6933] 00000080:00200000:1.0:1789760797.571272:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.571279:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.571296:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28401664 i_blocks 57353 00000020:00200000:1.0:1789760797.571302:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.571314:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.571347:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28401664, count: 4096 00000080:00200000:1.0:1789760797.571355:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28401664, bytes: 4096 00000080:00200000:1.0:1789760797.571369:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.571378:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.571380:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28401664, 28405760) 0 0 00000080:00200000:1.0:1789760797.571385:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.571402:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6934, 6934] 00020000:00200000:1.0:1789760797.571412:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28401664 chunk: [28401664, 28405760) 28405760, 4096 00020000:00200000:1.0:1789760797.571417:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.571438:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6934, 6934] 00000008:00010000:1.0:1789760797.572858:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.572875:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.572900:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28401664, 28405760) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.572919:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28401664=0x1b16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.573003:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.573007:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28405760 00000020:00010000:1.0:1789760797.573016:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.573019:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.573023:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.573026:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.573030:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.573040:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.573760:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.573802:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79cc0 00000008:00100000:1.0:1789760797.573958:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d500 - object 0x240000400:4 offset 28401664<>28405760 00000100:00100000:0.0:1789760797.574053:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495972096:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.610801:0:2356:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974250 gen 2 00000100:00100000:0.0:1789760797.610846:0:2356:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dd20504d500 x1876699495972096/t4294974251(4294974251) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760813 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760797.610894:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79cc0 00000020:00200000:0.0:1789760797.610902:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.610912:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.610934:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79cc0 00000100:00100000:0.0:1789760797.610959:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495972096:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.611932:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6934, 6934] 00000080:00200000:1.0:1789760797.611952:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.611959:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.611976:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28405760 i_blocks 57353 00000020:00200000:1.0:1789760797.611981:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.611991:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.612022:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28405760, count: 4096 00000080:00200000:1.0:1789760797.612030:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28405760, bytes: 4096 00000080:00200000:1.0:1789760797.612042:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.612049:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.612051:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28405760, 28409856) 0 0 00000080:00200000:1.0:1789760797.612055:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.612070:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6935, 6935] 00020000:00200000:1.0:1789760797.612077:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28405760 chunk: [28405760, 28409856) 28409856, 4096 00020000:00200000:1.0:1789760797.612081:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.612100:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6935, 6935] 00000008:00010000:1.0:1789760797.612117:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.612122:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.612132:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28405760, 28409856) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.612146:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28405760=0x1b17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.612262:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.612265:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28409856 00000020:00010000:1.0:1789760797.612273:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.612276:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.612280:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.612283:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.612286:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.612293:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.612300:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.612321:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79b40 00000008:00100000:1.0:1789760797.615810:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504d180 - object 0x240000400:4 offset 28405760<>28409856 00000100:00100000:1.0:1789760797.615925:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495972224:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.664365:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974252 gen 2 00000008:00010000:0.0:1789760797.664416:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79b40 00000020:00200000:0.0:1789760797.664423:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.664432:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.664459:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79b40 00000100:00100000:0.0:1789760797.664487:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495972224:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760797.664639:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6935, 6935] 00000080:00200000:2.0:1789760797.664652:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.664657:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.664669:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28409856 i_blocks 57353 00000020:00200000:2.0:1789760797.664673:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.664682:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.664707:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28409856, count: 4096 00000080:00200000:2.0:1789760797.664713:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28409856, bytes: 4096 00000080:00200000:2.0:1789760797.664724:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.664731:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.664733:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28409856, 28413952) 0 0 00000080:00200000:2.0:1789760797.664737:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.664749:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6936, 6936] 00020000:00200000:2.0:1789760797.664756:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28409856 chunk: [28409856, 28413952) 28413952, 4096 00020000:00200000:2.0:1789760797.664760:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.664775:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6936, 6936] 00000008:00010000:2.0:1789760797.664790:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.664796:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760797.664805:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28409856, 28413952) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.664815:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28409856=0x1b18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.664859:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.664862:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28413952 00000020:00010000:2.0:1789760797.664869:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.664872:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.664875:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.664878:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.664881:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.664887:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.664893:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.664906:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813a80 00000008:00100000:2.0:1789760797.665002:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205265880 - object 0x240000400:4 offset 28409856<>28413952 00000100:00100000:3.0:1789760797.665077:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495972352:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760797.710639:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974253 gen 2 00000008:00010000:3.0:1789760797.710676:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813a80 00000020:00200000:3.0:1789760797.710682:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.710690:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.710710:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813a80 00000100:00100000:3.0:1789760797.710734:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495972352:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760797.710832:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6936, 6936] 00000080:00200000:2.0:1789760797.710845:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.710850:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.710863:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28413952 i_blocks 57353 00000020:00200000:2.0:1789760797.710866:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.710874:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.710902:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28413952, count: 4096 00000080:00200000:2.0:1789760797.710909:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28413952, bytes: 4096 00000080:00200000:2.0:1789760797.710918:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.710925:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.710927:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28413952, 28418048) 0 0 00000080:00200000:2.0:1789760797.710931:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.710945:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6937, 6937] 00020000:00200000:2.0:1789760797.710952:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28413952 chunk: [28413952, 28418048) 28418048, 4096 00020000:00200000:2.0:1789760797.710955:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.710970:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6937, 6937] 00000008:00010000:2.0:1789760797.710984:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.710989:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760797.710998:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28413952, 28418048) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.711008:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28413952=0x1b19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.711053:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.711055:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28418048 00000020:00010000:2.0:1789760797.711061:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.711065:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.711068:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.711071:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.711073:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.711079:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.711083:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.711094:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813e40 00000008:00100000:2.0:1789760797.713365:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205264e00 - object 0x240000400:4 offset 28413952<>28418048 00000100:00100000:3.0:1789760797.713433:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.713467:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495972480:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760797.750736:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974254 gen 2 00000008:00010000:3.0:1789760797.750786:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813e40 00000020:00200000:3.0:1789760797.750793:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.750802:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.750821:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813e40 00000100:00100000:3.0:1789760797.750844:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495972480:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.753285:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6937, 6937] 00000080:00200000:1.0:1789760797.753305:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.753310:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.753323:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28418048 i_blocks 57353 00000020:00200000:1.0:1789760797.753327:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.753340:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.753367:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28418048, count: 4096 00000080:00200000:1.0:1789760797.753373:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28418048, bytes: 4096 00000080:00200000:1.0:1789760797.753385:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.753393:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.753395:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28418048, 28422144) 0 0 00000080:00200000:1.0:1789760797.753399:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.753411:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6938, 6938] 00020000:00200000:1.0:1789760797.753419:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28418048 chunk: [28418048, 28422144) 28422144, 4096 00020000:00200000:1.0:1789760797.753423:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.753439:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6938, 6938] 00000008:00010000:1.0:1789760797.753453:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.753459:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.753468:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28418048, 28422144) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.753479:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28418048=0x1b1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.753533:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.753536:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28422144 00000020:00010000:1.0:1789760797.753543:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.753546:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.753548:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.753551:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.753554:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.753560:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.753566:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.753583:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79840 00000008:00100000:1.0:1789760797.753683:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504e680 - object 0x240000400:4 offset 28418048<>28422144 00000100:00100000:0.0:1789760797.754492:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760797.754518:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495972608:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.808010:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974255 gen 2 00000008:00010000:0.0:1789760797.808048:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79840 00000020:00200000:0.0:1789760797.808056:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.808065:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.808082:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79840 00000100:00100000:0.0:1789760797.808105:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495972608:192.168.203.109@tcp:4:dd.0 00000080:00200000:1.0:1789760797.808206:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6938, 6938] 00000080:00200000:1.0:1789760797.808218:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760797.808223:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760797.808236:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28422144 i_blocks 57353 00000020:00200000:1.0:1789760797.808239:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760797.808247:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760797.808271:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28422144, count: 4096 00000080:00200000:1.0:1789760797.808276:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28422144, bytes: 4096 00000080:00200000:1.0:1789760797.808282:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760797.808289:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760797.808291:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28422144, 28426240) 0 0 00000080:00200000:1.0:1789760797.808294:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760797.808307:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6939, 6939] 00020000:00200000:1.0:1789760797.808313:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28422144 chunk: [28422144, 28426240) 28426240, 4096 00020000:00200000:1.0:1789760797.808317:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760797.808334:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6939, 6939] 00000008:00010000:1.0:1789760797.808348:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760797.808354:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20549b930, osc lock ffff8dd205030980, flags 0x0 00000080:00200000:1.0:1789760797.808362:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28422144, 28426240) append 0 append lockpos 0 00000080:00200000:1.0:1789760797.808371:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28422144=0x1b1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760797.808415:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760797.808418:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28426240 00000020:00010000:1.0:1789760797.808423:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760797.808426:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.808429:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20549b930 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760797.808433:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760797.808435:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20549b930 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760797.808440:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760797.808445:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760797.808461:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd219d79180 00000008:00100000:1.0:1789760797.809705:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd20504c700 - object 0x240000400:4 offset 28422144<>28426240 00000100:00100000:0.0:1789760797.809770:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495972736:192.168.203.109@tcp:4:dd.0 00000100:00100000:0.0:1789760797.855515:0:2357:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974256 gen 2 00000008:00010000:0.0:1789760797.855566:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd219d79180 00000020:00200000:0.0:1789760797.855572:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.855581:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.855602:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd219d79180 00000100:00100000:0.0:1789760797.855631:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495972736:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760797.855648:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6939, 6939] 00000080:00200000:3.0:1789760797.855742:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.855750:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.855767:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28426240 i_blocks 57353 00000020:00200000:3.0:1789760797.855772:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.855784:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.855819:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28426240, count: 4096 00000080:00200000:3.0:1789760797.855827:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28426240, bytes: 4096 00000080:00200000:3.0:1789760797.855837:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.855846:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.855905:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28426240, 28430336) 0 0 00000080:00200000:3.0:1789760797.855910:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.855929:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6940, 6940] 00020000:00200000:3.0:1789760797.855938:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28426240 chunk: [28426240, 28430336) 28430336, 4096 00020000:00200000:3.0:1789760797.855942:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.855964:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6940, 6940] 00000008:00010000:3.0:1789760797.855983:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.855989:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760797.855999:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28426240, 28430336) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.856013:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28426240=0x1b1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.856077:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.856080:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28430336 00000020:00010000:3.0:1789760797.856089:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.856093:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.856096:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.856100:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.856103:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.856111:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.856117:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.856136:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760797.856380:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 28426240<>28430336 00000100:00100000:3.0:1789760797.856438:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495972864:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760797.900154:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974257 gen 2 00000008:00010000:3.0:1789760797.900251:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd2082503c0 00000020:00200000:3.0:1789760797.900259:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.900269:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.900291:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd2082503c0 00000100:00100000:3.0:1789760797.900318:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495972864:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760797.900567:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6940, 6940] 00000080:00200000:2.0:1789760797.900586:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.900592:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.900606:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28430336 i_blocks 57353 00000020:00200000:2.0:1789760797.900610:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.900623:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.900658:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28430336, count: 4096 00000080:00200000:2.0:1789760797.900666:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28430336, bytes: 4096 00000080:00200000:2.0:1789760797.900676:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.900684:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.900686:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28430336, 28434432) 0 0 00000080:00200000:2.0:1789760797.900690:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.900751:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6941, 6941] 00020000:00200000:2.0:1789760797.900759:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28430336 chunk: [28430336, 28434432) 28434432, 4096 00020000:00200000:2.0:1789760797.900763:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.900784:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6941, 6941] 00000008:00010000:2.0:1789760797.900855:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.900861:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760797.900870:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28430336, 28434432) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.900881:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28430336=0x1b1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.900933:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.900935:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28434432 00000020:00010000:2.0:1789760797.900942:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.900945:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.900948:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.900951:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.900953:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.900959:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.900964:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.900985:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813180 00000008:00100000:2.0:1789760797.901122:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267800 - object 0x240000400:4 offset 28430336<>28434432 00000100:00100000:3.0:1789760797.901816:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.901848:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495972992:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760797.981502:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974258 gen 2 00000008:00010000:3.0:1789760797.981553:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813180 00000020:00200000:3.0:1789760797.981560:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.981572:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.981600:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813180 00000100:00100000:3.0:1789760797.981626:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495972992:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760797.984061:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6941, 6941] 00000080:00200000:2.0:1789760797.984090:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.984158:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.984222:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28434432 i_blocks 57353 00000020:00200000:2.0:1789760797.984228:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.984243:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.984278:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28434432, count: 4096 00000080:00200000:2.0:1789760797.984287:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28434432, bytes: 4096 00000080:00200000:2.0:1789760797.984302:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.984311:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.984314:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28434432, 28438528) 0 0 00000080:00200000:2.0:1789760797.984318:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.984333:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6942, 6942] 00020000:00200000:2.0:1789760797.984342:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28434432 chunk: [28434432, 28438528) 28438528, 4096 00020000:00200000:2.0:1789760797.984348:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.984370:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6942, 6942] 00000008:00010000:2.0:1789760797.984389:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.984394:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760797.984405:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28434432, 28438528) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.984420:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28434432=0x1b1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.984491:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.984494:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28438528 00000020:00010000:2.0:1789760797.984519:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.984522:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.984525:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.984529:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.984531:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.984540:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.984546:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.984566:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813f00 00000008:00100000:2.0:1789760797.984713:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267100 - object 0x240000400:4 offset 28434432<>28438528 00000100:00100000:3.0:1789760797.984821:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495973120:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760798.020384:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974259 gen 2 00000008:00010000:3.0:1789760798.020435:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813f00 00000020:00200000:3.0:1789760798.020444:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760798.020456:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760798.020481:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813f00 00000100:00100000:3.0:1789760798.020518:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495973120:192.168.203.109@tcp:4:dd.0 00000080:00200000:2.0:1789760798.022552:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6942, 6942] 00000080:00200000:2.0:1789760798.022581:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760798.022588:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760798.022607:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28438528 i_blocks 57353 00000020:00200000:2.0:1789760798.022612:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760798.022627:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760798.022659:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28438528, count: 4096 00000080:00200000:2.0:1789760798.022667:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28438528, bytes: 4096 00000080:00200000:2.0:1789760798.022681:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760798.022691:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760798.022694:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28438528, 28442624) 0 0 00000080:00200000:2.0:1789760798.022698:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760798.022714:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6943, 6943] 00020000:00200000:2.0:1789760798.022721:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28438528 chunk: [28438528, 28442624) 28442624, 4096 00020000:00200000:2.0:1789760798.022726:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760798.022749:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6943, 6943] 00000008:00010000:2.0:1789760798.022767:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760798.022773:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20529d430, osc lock ffff8dd205d04260, flags 0x0 00000080:00200000:2.0:1789760798.022786:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28438528, 28442624) append 0 append lockpos 0 00000080:00200000:2.0:1789760798.022802:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28438528=0x1b1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760798.022883:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760798.022886:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28442624 00000020:00010000:2.0:1789760798.022896:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760798.022900:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760798.022903:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20529d430 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760798.022907:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760798.022910:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20529d430 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760798.022919:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760798.022927:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760798.022948:0:256:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd208813d80 00000008:00100000:2.0:1789760798.023085:0:256:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd205267b80 - object 0x240000400:4 offset 28438528<>28442624 00000100:00100000:3.0:1789760798.027443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495973248:192.168.203.109@tcp:4:dd.0 00000100:00100000:3.0:1789760798.089535:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dd2055fc800: committing for last_committed 4294974260 gen 2 00000008:00010000:3.0:1789760798.089592:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd208813d80 00000020:00200000:3.0:1789760798.089600:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760798.089612:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760798.089633:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd208813d80 00000100:00100000:3.0:1789760798.089665:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495973248:192.168.203.109@tcp:4:dd.0 00000080:00200000:0.0:1789760798.090116:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6943, 6943] 00000080:00200000:0.0:1789760798.090134:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760798.090141:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760798.090157:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28442624 i_blocks 57353 00000020:00200000:0.0:1789760798.090161:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760798.090210:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760798.090247:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28442624, count: 4096 00000080:00200000:0.0:1789760798.090254:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28442624, bytes: 4096 00000080:00200000:0.0:1789760798.090266:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760798.090276:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760798.090278:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28442624, 28446720) 0 0 00000080:00200000:0.0:1789760798.090283:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760798.090300:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6944, 6944] 00020000:00200000:0.0:1789760798.090307:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28442624 chunk: [28442624, 28446720) 28446720, 4096 00020000:00200000:0.0:1789760798.090312:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760798.090331:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6944, 6944] 00000008:00010000:0.0:1789760798.090349:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760798.090354:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd20931d4b0, osc lock ffff8dd205cca390, flags 0x0 00000080:00200000:0.0:1789760798.090366:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28442624, 28446720) append 0 append lockpos 0 00000080:00200000:0.0:1789760798.090379:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28442624=0x1b20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760798.090438:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760798.090441:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28446720 00000020:00010000:0.0:1789760798.090450:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760798.090454:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.090457:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd20931d4b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760798.090460:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760798.090463:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd20931d4b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760798.090470:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760798.090477:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760798.090494:0:366:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd20845ea80 00000008:00100000:0.0:1789760798.090640:0:366:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd211835500 - object 0x240000400:4 offset 28442624<>28446720 00000100:00100000:0.0:1789760798.090691:0:2357:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760798.090710:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495973376:192.168.203.109@tcp:4:dd.0 00000008:00010000:0.0:1789760798.132717:0:2357:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dd20845ea80 00000020:00200000:0.0:1789760798.132733:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760798.132744:0:2357:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760798.132774:0:2357:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dd20845ea80 00000100:00100000:0.0:1789760798.132808:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495973376:192.168.203.109@tcp:4:dd.0 00000080:00200000:3.0:1789760798.132915:0:11747:0:(file.c:2205:ll_file_io_generic()) Range unlock [6944, 6944] 00000080:00200000:3.0:1789760798.132932:0:11747:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760798.132939:0:11747:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760798.132954:0:11747:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28446720 i_blocks 57353 00000020:00200000:3.0:1789760798.132958:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760798.132968:0:11747:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760798.133003:0:11747:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28446720, count: 4096 00000080:00200000:3.0:1789760798.133010:0:11747:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28446720, bytes: 4096 00000080:00200000:3.0:1789760798.133021:0:11747:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760798.133030:0:11747:0:(cl_io.c:214:cl_io_rw_init()) header@000000009f148b39[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760798.133032:0:11747:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28446720, 28450816) 0 0 00000080:00200000:3.0:1789760798.133036:0:11747:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760798.133053:0:11747:0:(file.c:2168:ll_file_io_generic()) Range lock [6945, 6945] 00020000:00200000:3.0:1789760798.133061:0:11747:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28446720 chunk: [28446720, 28450816) 28450816, 4096 00020000:00200000:3.0:1789760798.133066:0:11747:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760798.133086:0:11747:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6945, 6945] 00000008:00010000:3.0:1789760798.133104:0:11747:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760798.133109:0:11747:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dd2203ac1b0, osc lock ffff8dd205052be0, flags 0x0 00000080:00200000:3.0:1789760798.133122:0:11747:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28446720, 28450816) append 0 append lockpos 0 00000080:00200000:3.0:1789760798.133136:0:11747:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dd211126c08), size=4096 (max 536870912), offset=28446720=0x1b21000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760798.133236:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760798.133239:0:11747:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28450816 00000020:00010000:3.0:1789760798.133248:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760798.133252:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760798.133254:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dd2203ac1b0 (ffff8dd205fe0638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760798.133257:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd208814b78 (ffff8dd205fe0908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760798.133260:0:11747:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dd2203ac1b0 (ffff8dd205fe0908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760798.133267:0:11747:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760798.133274:0:11747:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760798.133290:0:11116:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dd2082503c0 00000008:00100000:3.0:1789760798.133414:0:11116:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dd203146680 - object 0x240000400:4 offset 28446720<>28450816 00000100:00100000:2.0:1789760798.133496:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495973504:192.168.203.109@tcp:4:dd.0 00010000:00010000:0.0:1789760798.155451:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760799.179422:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760799.630607:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2359:1876699495973632:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760799.631297:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760799.631309:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495973760:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760799.632281:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495974016:192.168.203.109@tcp:400:kworker.0 00000100:00100000:2.0:1789760799.632372:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760799.632379:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495973888:192.168.203.109@tcp:400:kworker.0 00010000:00010000:0.0:1789760800.203452:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760801.227608:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760802.251442:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760803.275385:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760804.299411:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760804.747421:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495974272:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760804.747436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495974144:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760804.747500:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495974528:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760804.747507:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495974400:192.168.203.109@tcp:400:kworker.0 00010000:00010000:0.0:1789760805.323340:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760806.347367:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760807.371317:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760808.395289:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760809.419301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760809.867343:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495974656:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760809.867358:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495974784:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760809.867430:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495974912:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760809.867435:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495975040:192.168.203.109@tcp:400:kworker.0 00010000:00010000:0.0:1789760810.443350:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760811.467337:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760812.491367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760813.515364:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760814.539365:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1789760814.987286:0:2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760798/real 1789760798] req@ffff8dd203146680 x1876699495973504/t0(0) o4->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789760814 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1789760814.987355:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495975296:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760814.987365:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495975168:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760814.987467:0:2356:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495975552:192.168.203.109@tcp:400:kworker.0 00000100:00100000:1.0:1789760814.987469:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495975424:192.168.203.109@tcp:400:kworker.0 00000100:00080000:3.0:1789760814.995507:0:2358:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1789760814.995516:0:2358:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8dd2055fc800: Connection to lustre-OST0000 (at 192.168.203.109@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1789760814.999940:0:2358:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1789760815.000006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760815.000014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760815.000018:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760815.000026:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 373 00000100:00080000:0.0:1789760815.000041:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760815.000055:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760815.000107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699495975680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760815.000193:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495975680:192.168.203.109@tcp:8:kworker.0 00010000:00010000:2.0:1789760815.563339:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1789760816.011260:0:2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760799/real 1789760799] req@ffff8dd203144380 x1876699495973888/t0(0) o400->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1789760816.011310:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760799/real 1789760799] req@ffff8dd203146300 x1876699495973760/t0(0) o400->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760816.011363:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd203146300 x1876699495973760/t0(0) o400->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760816.011375:0:2359:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1789760816.011381:0:2359:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8dd2055fc800: Connection to lustre-MDT0000 (at 192.168.203.109@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:3.0:1789760816.018792:0:2358:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd203144380 x1876699495973888/t0(0) o400->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760816.018807:0:2358:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8dd20509e000 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:3.0:1789760816.018813:0:2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760799/real 1789760799] req@ffff8dd203145180 x1876699495974016/t0(0) o400->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760816.018822:0:2358:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd203145180 x1876699495974016/t0(0) o400->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760816.018828:0:2358:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1789760816.018831:0:2358:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8dd203049000: Connection to lustre-MDT0000 (at 192.168.203.109@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1789760816.018834:0:2358:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00100000:3.0:1789760816.018855:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:17022965-1e4f-46fb-915c-61730096b8e2:2358:1876699495973888:192.168.203.109@tcp:400:kworker.0 00000100:00100000:3.0:1789760816.018862:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2358:1876699495974016:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760816.018957:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495975808:192.168.203.109@tcp:400:kworker.0 00000100:00080000:1.0:1789760816.030224:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760816.030227:0:2359:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:1.0:1789760816.030238:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760816.030242:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760816.030251:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 236 00000100:00080000:1.0:1789760816.030264:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760816.030277:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760816.030303:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699495975936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760816.030329:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760816.030335:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:2.0:1789760816.030335:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760799/real 1789760799] req@ffff8dd203145c00 x1876699495973632/t0(0) o400->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760816.030344:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760816.030357:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789760816.030362:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495975936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1789760816.030362:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760816.030380:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 299 00000100:00080000:1.0:1789760816.030388:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760816.030391:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760816.030406:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699495976064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760816.030412:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd203145c00 x1876699495973632/t0(0) o400->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760816.030439:0:2359:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1789760816.030445:0:2359:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.203.109@tcp: Connection to MGS (at 192.168.203.109@tcp) was lost; in progress operations using this service will fail 00000100:00100000:0.0:1789760816.030456:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495976064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1789760816.034636:0:2359:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.203.109@tcp for MGC192.168.203.109@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1789760816.034640:0:2359:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1789760816.034647:0:2359:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dd203145c00 x1876699495973632/t0(0) o400->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760815 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760816.034661:0:2359:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dd211834000 x1876699495974144/t0(0) o400->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760820 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760816.034680:0:2359:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dd211836a00 x1876699495974656/t0(0) o400->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760825 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760816.034690:0:2359:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dd211835c00 x1876699495975168/t0(0) o400->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760830 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760816.034701:0:2359:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dd20504e300 x1876699495975808/t0(0) o400->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760832 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760816.034722:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:2.0:1789760816.034724:0:2359:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1789760816.034734:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760816.034734:0:2359:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 00000100:00080000:1.0:1789760816.034741:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:2.0:1789760816.034741:0:2359:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1789760816.034744:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495975808:192.168.203.109@tcp:400:kworker.0 00000100:00080000:2.0:1789760816.034749:0:2359:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1789760816.034750:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1789760816.034775:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495975168:192.168.203.109@tcp:400:kworker.0 00000100:00080000:1.0:1789760816.034784:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789760816.034794:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:17022965-1e4f-46fb-915c-61730096b8e2:2359:1876699495973760:192.168.203.109@tcp:400:kworker.0 00000100:00080000:1.0:1789760816.034795:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760816.034801:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789760816.034813:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2359:1876699495973632:192.168.203.109@tcp:400:kworker.0 00000100:00080000:1.0:1789760816.034814:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 236 00000100:00100000:0.0:1789760816.034824:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495974656:192.168.203.109@tcp:400:kworker.0 00000100:00080000:1.0:1789760816.034833:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789760816.034856:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699495976192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760816.034858:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:5292164f-12cc-4fea-b062-6f9c12f1be43:2357:1876699495974144:192.168.203.109@tcp:400:kworker.0 00000100:00080000:1.0:1789760816.034890:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760816.034895:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760816.034899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760816.034902:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760816.034928:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495976192:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789760816.587352:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760817.611393:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760818.635371:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760819.659315:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760820.043272:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760820.043281:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760820.043284:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760820.043287:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760820.043289:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760820.043291:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760820.043293:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760820.043295:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:0.0:1789760820.107262:0:2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760804/real 1789760804] req@ffff8dd211836680 x1876699495974272/t0(0) o400->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1789760820.107539:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760804/real 1789760804] req@ffff8dd211834e00 x1876699495974400/t0(0) o400->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760820.107586:0:2357:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211834e00 x1876699495974400/t0(0) o400->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760820 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760820.107593:0:2357:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8dd20509e000 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:1.0:1789760820.107611:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495974400:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760820.122492:0:2356:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211836680 x1876699495974272/t0(0) o400->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760820 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760820.122503:0:2356:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dd20509b000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1789760820.122512:0:2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760804/real 1789760804] req@ffff8dd211835880 x1876699495974528/t0(0) o400->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760820.122527:0:2356:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211835880 x1876699495974528/t0(0) o400->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760820 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760820.122532:0:2356:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dd20313b000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1789760820.122547:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495974272:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760820.122551:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495974528:192.168.203.109@tcp:400:kworker.0 00010000:00010000:0.0:1789760820.683348:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760821.067299:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760821.067310:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760821.067313:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760821.067317:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760821.067319:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760821.067322:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760821.067325:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760821.067328:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760821.707337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760822.731348:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760823.755356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760824.779411:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760825.163285:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760815/real 1789760815] req@ffff8dd211836d80 x1876699495975680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760825.163327:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699495975680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760825 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760825.163352:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760825.163382:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760825.163387:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495975680:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1789760825.163421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760825.163427:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760825.163431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760825.163434:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760825.163435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760825.163437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760825.163440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760825.163447:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 799 00000100:00080000:0.0:1789760825.163459:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1789760825.163466:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760825.163477:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760825.163511:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835880 x1876699495976320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760825.163536:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760825.163539:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760825.163574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495976320:192.168.203.109@tcp:8:kworker.0 00000100:00000400:0.0:1789760825.227291:0:2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760809/real 1789760809] req@ffff8dd211834a80 x1876699495974784/t0(0) o400->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1789760825.227291:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760809/real 1789760809] req@ffff8dd211835180 x1876699495974912/t0(0) o400->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760825.227353:0:2357:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211835180 x1876699495974912/t0(0) o400->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760825 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760825.227363:0:2357:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8dd20509e000 already not connected (conn 4, was 2): CONNECTING 00000100:00100000:1.0:1789760825.227383:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495974912:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760825.245807:0:2356:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211834a80 x1876699495974784/t0(0) o400->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760825 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760825.245819:0:2356:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dd20509b000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1789760825.245840:0:2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760809/real 1789760809] req@ffff8dd211834380 x1876699495975040/t0(0) o400->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760825.245851:0:2356:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211834380 x1876699495975040/t0(0) o400->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760825 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760825.245857:0:2356:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dd20313b000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1789760825.245871:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495974784:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760825.245876:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495975040:192.168.203.109@tcp:400:kworker.0 00010000:00010000:0.0:1789760825.803382:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760826.187368:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760816/real 1789760816] req@ffff8dd20504d180 x1876699495976192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760826.187430:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699495976192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760826.187445:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760816/real 1789760816] req@ffff8dd20504e680 x1876699495976064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760826.187463:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699495976064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760826.187478:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760816/real 1789760816] req@ffff8dd20504c700 x1876699495975936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760826.187494:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699495975936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760826.187523:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760826.187562:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760826.187569:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495976192:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789760826.187578:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760826.187590:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760826.187595:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495976064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1789760826.187604:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760826.187605:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760826.187625:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760826.187630:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760826.187638:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 800 00000100:00080000:3.0:1789760826.187642:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760826.187650:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495975936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1789760826.187655:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1789760826.187668:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760826.187722:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699495976448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760826.187753:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760826.187758:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760826.187761:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760826.187766:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 800 00000100:00080000:0.0:1789760826.187773:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1789760826.187780:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760826.187795:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760826.187803:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495976448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760826.187817:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699495976576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760826.187828:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760826.187832:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760826.187835:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760826.187838:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760826.187840:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760826.187843:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 800 00000100:00080000:0.0:1789760826.187847:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1789760826.187850:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760826.187854:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760826.187865:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699495976704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760826.187880:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760826.187883:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760826.187944:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495976576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760826.188020:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495976704:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760826.827356:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789760827.275444:0:468:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1789760827.275470:0:468:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760827.275530:0:468:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1789760827.851341:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760828.875445:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760829.899481:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760830.219326:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760830.219338:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760830.219343:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760830.219347:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760830.219351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760830.219354:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760830.219358:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760830.219362:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:0.0:1789760830.347336:0:2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff8dd211834700 x1876699495975296/t0(0) o400->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1789760830.347336:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff8dd211837480 x1876699495975424/t0(0) o400->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760830.347431:0:2357:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211837480 x1876699495975424/t0(0) o400->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760830.347446:0:2357:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8dd20509e000 already not connected (conn 4, was 2): CONNECTING 00000100:00100000:1.0:1789760830.347469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:17022965-1e4f-46fb-915c-61730096b8e2:2357:1876699495975424:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760830.370019:0:2356:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211834700 x1876699495975296/t0(0) o400->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760830.370029:0:2356:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dd20509b000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:0.0:1789760830.370037:0:2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff8dd211835f80 x1876699495975552/t0(0) o400->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760830.370052:0:2356:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dd211835f80 x1876699495975552/t0(0) o400->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760830 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760830.370060:0:2356:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dd20313b000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1789760830.370072:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:17022965-1e4f-46fb-915c-61730096b8e2:2356:1876699495975296:192.168.203.109@tcp:400:kworker.0 00000100:00100000:0.0:1789760830.370078:0:2356:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2356:1876699495975552:192.168.203.109@tcp:400:kworker.0 00010000:00010000:0.0:1789760830.923416:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760831.243313:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760831.243333:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760831.243341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760831.243348:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760831.243353:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760831.243357:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760831.243364:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760831.243368:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1789760831.947456:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760832.971403:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760833.995408:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760835.019483:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760836.043395:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760836.491323:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760836.491337:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760836.491341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760836.491346:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760836.491349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760836.491354:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760836.491358:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760836.491363:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760837.067384:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760838.091413:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760839.115346:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760840.139440:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760840.459341:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760825/real 1789760825] req@ffff8dd211835880 x1876699495976320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760840.459394:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835880 x1876699495976320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760840 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760840.459430:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760840.459464:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760840.459481:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495976320:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1789760840.459533:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.459544:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760840.459548:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.459552:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760840.459556:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.459559:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760840.459563:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760840.459575:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 809 00000100:00080000:1.0:1789760840.459590:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1789760840.459598:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760840.459627:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760840.459669:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699495976832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760840.459715:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760840.459719:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760840.459770:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495976832:192.168.203.109@tcp:8:kworker.0 00010000:00010000:0.0:1789760841.163342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760841.483340:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760826/real 1789760826] req@ffff8dd211834a80 x1876699495976576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760841.483405:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699495976576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760841.483417:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760826/real 1789760826] req@ffff8dd211836680 x1876699495976704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760841.483430:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699495976704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760841.483443:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760826/real 1789760826] req@ffff8dd211834380 x1876699495976448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760841.483457:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699495976448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760841.483482:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760841.483509:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760841.483515:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495976576:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1789760841.483523:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760841.483547:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760841.483553:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495976704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1789760841.483560:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760841.483573:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760841.483575:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760841.483580:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495976448:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789760841.483581:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760841.483586:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760841.483594:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 810 00000100:00080000:1.0:1789760841.483610:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1789760841.483621:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789760841.483671:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699495976960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760841.483699:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760841.483704:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760841.483707:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760841.483713:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 810 00000100:00080000:1.0:1789760841.483719:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1789760841.483725:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760841.483739:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760841.483743:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495976960:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760841.483757:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699495977088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760841.483770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760841.483775:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760841.483778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760841.483782:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760841.483784:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760841.483788:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 810 00000100:00080000:1.0:1789760841.483792:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1789760841.483812:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760841.483818:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760841.483829:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699495977216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760841.483845:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760841.483849:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760841.483921:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495977088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760841.483972:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495977216:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760842.187412:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760843.211307:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760844.235332:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760845.259296:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760845.515232:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760845.515243:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760845.515246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760845.515250:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760845.515253:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760845.515255:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760845.515260:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760845.515263:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760846.283452:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760846.539332:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760846.539354:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760846.539361:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760846.539370:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760846.539377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760846.539385:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760846.539392:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760846.539398:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760847.307380:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760848.331310:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760849.355459:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760850.379444:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760851.403395:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760851.851320:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760851.851334:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760851.851338:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760851.851352:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760851.851355:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760851.851359:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760851.851364:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760851.851367:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1789760852.427411:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760853.451400:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760854.475461:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.1F:1789760855.435378:0:2347:0:(socklnd_cb.c:2294:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.203.109@tcp (192.168.203.109:988) the network or that node may be down. 00010000:00010000:0.0:1789760855.499396:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760856.523361:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760856.971276:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760856.971295:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760856.971301:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760856.971307:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760856.971313:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760856.971319:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760856.971325:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760856.971331:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760857.547333:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760858.571404:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760859.595333:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760860.619305:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760860.875275:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760840/real 1789760840] req@ffff8dd20504d500 x1876699495976832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760860 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760860.875302:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699495976832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760860 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760860.875319:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760860.875336:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760860.875340:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495976832:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1789760860.875402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760860.875407:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760860.875410:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760860.875412:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760860.875414:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760860.875415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760860.875418:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760860.875424:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 825 00000100:00080000:2.0:1789760860.875432:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760860.875439:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760860.875449:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760860.875481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699495977344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760860.875511:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760860.875515:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760860.875554:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495977344:192.168.203.109@tcp:8:kworker.0 00010000:00010000:0.0:1789760861.643346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760861.899295:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760841/real 1789760841] req@ffff8dd20504df80 x1876699495977088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760861.899334:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699495977088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760861.899349:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760841/real 1789760841] req@ffff8dd20504fb80 x1876699495977216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760861.899364:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699495977216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760861.899375:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760841/real 1789760841] req@ffff8dd20504f800 x1876699495976960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760861.899386:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699495976960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760861.899407:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760861.899437:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760861.899443:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495977088:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1789760861.899450:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760861.899456:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760861.899459:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495977216:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1789760861.899464:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760861.899468:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760861.899471:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495976960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789760861.899501:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760861.899522:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760861.899527:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760861.899537:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 826 00000100:00080000:2.0:1789760861.899550:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760861.899559:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789760861.899599:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699495977472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760861.899636:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760861.899640:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760861.899642:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760861.899646:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 826 00000100:00080000:2.0:1789760861.899650:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760861.899657:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760861.899668:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760861.899684:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699495977600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760861.899696:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495977472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789760861.899727:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760861.899745:0:468:0:(pinger.c:247:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760861.899748:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760861.899751:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760861.899753:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760861.899759:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 826 00000100:00080000:2.0:1789760861.899765:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760861.899770:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760861.899776:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760861.899793:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699495977728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760861.899897:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495977600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760861.899938:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495977728:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760862.667303:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789760863.179475:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.109:988 00000400:00000100:1.0:1789760863.179538:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760863.179559:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760863.179625:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760863.179637:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760863.179647:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789760863.179690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760861/real 1789760863] req@ffff8dd205264a80 x1876699495977600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.179740:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699495977600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760863.179765:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760863.179771:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760863.179776:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495977600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760863.179785:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760861/real 1789760863] req@ffff8dd205267480 x1876699495977728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.179800:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699495977728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760863.179808:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760863.179812:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760863.179816:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495977728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760863.179823:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760861/real 1789760863] req@ffff8dd205266680 x1876699495977472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.179834:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699495977472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760863.179842:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760863.179846:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760863.179849:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495977472:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789760863.179855:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760860/real 1789760863] req@ffff8dd205265500 x1876699495977344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760863.179867:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699495977344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760863.179877:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760863.179880:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760863.179884:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495977344:192.168.203.109@tcp:8:kworker.0 00010000:00010000:0.0:1789760863.691328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760864.715330:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760865.739405:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760865.931320:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760865.931332:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760865.931337:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760865.931350:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 846 00000100:00080000:2.0:1789760865.931364:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760865.931373:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789760865.931420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699495977856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760865.931448:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760865.931452:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760865.931454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760865.931458:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 846 00000100:00080000:2.0:1789760865.931460:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760865.931463:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760865.931474:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760865.931484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699495977984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760865.931492:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760865.931495:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760865.931497:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760865.931501:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 845 00000100:00080000:2.0:1789760865.931504:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760865.931508:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760865.931512:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760865.931521:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699495978112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760865.931530:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760865.931533:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760865.931535:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760865.931541:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 846 00000100:00100000:3.0:1789760865.931543:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495977856:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789760865.931544:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760865.931579:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760865.931587:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760865.931602:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699495978240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760865.931812:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495977984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760865.931873:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495978112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789760865.931902:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495978240:192.168.203.109@tcp:38:kworker.0 00000400:00000100:1.0:1789760866.251491:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.109:988 00000400:00000100:1.0:1789760866.251550:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760866.251570:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760866.251632:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760866.251640:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760866.251647:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789760866.251690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760865/real 1789760866] req@ffff8dd205267480 x1876699495978112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760866.251714:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699495978112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760866.251724:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760866.251728:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760866.251732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495978112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789760866.251738:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760865/real 1789760866] req@ffff8dd205264a80 x1876699495978240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760866.251746:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699495978240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760866.251752:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760866.251755:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760866.251757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495978240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760866.251762:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760865/real 1789760866] req@ffff8dd205265500 x1876699495977856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760866.251771:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699495977856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760866.251776:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760866.251778:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760866.251780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495977856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789760866.251784:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760865/real 1789760866] req@ffff8dd205266680 x1876699495977984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760866.251791:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699495977984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760866.251796:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760866.251798:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760866.251800:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495977984:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760866.763428:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760867.787372:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760868.811301:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760869.835327:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760870.859331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760871.307354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760871.307363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760871.307368:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760871.307393:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 850 00000100:00080000:3.0:1789760871.307405:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760871.307411:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760871.307446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699495978368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760871.307505:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760871.307508:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760871.307510:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760871.307515:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 850 00000100:00080000:3.0:1789760871.307518:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760871.307522:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760871.307533:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760871.307545:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699495978496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760871.307558:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495978368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789760871.307559:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760871.307570:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760871.307589:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760871.307593:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 850 00000100:00080000:3.0:1789760871.307596:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760871.307599:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760871.307613:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760871.307624:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699495978624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760871.307631:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760871.307634:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760871.307732:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760871.307737:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 850 00000100:00080000:3.0:1789760871.307741:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789760871.307744:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495978496:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1789760871.307747:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760871.307770:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760871.307786:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699495978752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760871.307800:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495978624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760871.307833:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495978752:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760871.883327:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760872.907353:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789760873.227591:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.109:988 00000400:00000100:1.0:1789760873.227684:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760873.227712:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760873.227800:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760873.227856:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789760873.227859:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760871/real 1789760873] req@ffff8dd203145180 x1876699495978368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1789760873.227876:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789760873.227906:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699495978368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760873.227919:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760873.227925:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760873.227929:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495978368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760873.227949:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760871/real 1789760873] req@ffff8dd203145f80 x1876699495978752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760873.227961:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699495978752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760873.227967:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760873.227972:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760873.227976:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495978752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760873.227992:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760871/real 1789760873] req@ffff8dd203144380 x1876699495978496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760873.228009:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699495978496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760873.228019:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760873.228024:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760873.228029:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495978496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760873.228037:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760871/real 1789760873] req@ffff8dd203144a80 x1876699495978624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760873.228053:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699495978624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760873.228063:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760873.228067:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760873.228071:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495978624:192.168.203.109@tcp:8:kworker.0 00010000:00010000:0.0:1789760873.931365:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760874.955299:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760875.979291:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760876.427248:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760876.427256:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760876.427260:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760876.427277:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 856 00000100:00080000:1.0:1789760876.427286:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760876.427293:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789760876.427326:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699495978880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760876.427346:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760876.427349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760876.427350:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760876.427353:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 856 00000100:00080000:1.0:1789760876.427355:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760876.427357:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760876.427365:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760876.427372:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699495979008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760876.427379:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760876.427382:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760876.427383:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760876.427386:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 856 00000100:00080000:1.0:1789760876.427388:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760876.427391:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760876.427394:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760876.427400:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699495979136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760876.427407:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760876.427409:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760876.427410:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760876.427412:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 856 00000100:00080000:1.0:1789760876.427414:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760876.427417:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760876.427421:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760876.427423:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495978880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760876.427430:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699495979264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760876.427519:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495979008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760876.427545:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495979136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760876.427569:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495979264:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789760877.003348:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760878.027362:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760879.051318:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789760879.499355:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.109:988 00000400:00000100:3.0:1789760879.499401:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789760879.499422:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789760879.499467:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789760879.499481:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789760879.499493:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789760879.499505:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760876/real 1789760879] req@ffff8dd20504ce00 x1876699495978880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760879.499538:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699495978880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760879.499553:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760879.499559:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760879.499563:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495978880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760879.499573:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760876/real 1789760879] req@ffff8dd20504f100 x1876699495979008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760879.499600:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699495979008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760879.499609:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760879.499613:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760879.499616:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495979008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760879.499624:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760876/real 1789760879] req@ffff8dd20504ed80 x1876699495979136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760879.499636:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699495979136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760879.499644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760879.499648:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760879.499651:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495979136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760879.499683:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760876/real 1789760879] req@ffff8dd20504d880 x1876699495979264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760879.499703:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699495979264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760879.499712:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760879.499716:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760879.499719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495979264:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760880.075283:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760881.099276:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760881.547293:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760881.547303:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760881.547308:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760881.547318:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 861 00000100:00080000:1.0:1789760881.547330:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760881.547337:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789760881.547379:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699495979392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760881.547432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760881.547436:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760881.547438:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760881.547443:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 861 00000100:00080000:1.0:1789760881.547448:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760881.547452:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760881.547465:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760881.547489:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699495979520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760881.547491:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495979392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789760881.547522:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760881.547528:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760881.547533:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760881.547543:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 861 00000100:00080000:1.0:1789760881.547587:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760881.547600:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789760881.547610:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789760881.547627:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760881.547651:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699495979648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760881.547666:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760881.547670:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760881.547672:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760881.547680:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 861 00000100:00100000:0.0:1789760881.547682:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760881/real 1789760881] req@ffff8dd20504c000 x1876699495979392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760881.547689:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760881.547701:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760881.547711:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760881.547726:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699495979776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760881.547728:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699495979392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760881.547765:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760881.547773:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760881.547779:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495979392:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760881.547810:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495979520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760881.547839:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760881.547854:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760881/real 1789760881] req@ffff8dd20504ea00 x1876699495979520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760881.547867:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699495979520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760881.547882:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760881.547887:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760881.547891:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495979520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760881.547898:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495979648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760881.547921:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760881.547935:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760881/real 1789760881] req@ffff8dd20504c380 x1876699495979648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760881.547947:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699495979648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760881.547956:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760881.547960:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760881.547963:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495979648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760881.547969:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495979776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760881.547991:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760881.548003:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760881/real 1789760881] req@ffff8dd20504dc00 x1876699495979776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760881.548014:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699495979776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760881.548021:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760881.548025:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760881.548027:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495979776:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760882.123315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760883.147291:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760884.171279:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760885.195302:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760886.219293:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760886.667295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760886.667304:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760886.667307:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760886.667316:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 866 00000100:00080000:3.0:1789760886.667325:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760886.667332:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760886.667361:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699495979904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760886.667384:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760886.667387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760886.667388:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760886.667391:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 866 00000100:00080000:3.0:1789760886.667394:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760886.667397:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760886.667405:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760886.667414:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699495980032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760886.667422:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495979904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789760886.667424:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760886.667432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760886.667433:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760886.667437:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 866 00000100:00080000:3.0:1789760886.667439:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760886.667441:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760886.667445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760886.667452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699495980160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760886.667459:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760886.667462:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760886.667463:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760886.667465:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 866 00000100:00080000:3.0:1789760886.667468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760886.667470:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760886.667473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760886.667480:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699495980288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789760886.667492:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760886.667542:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760886/real 1789760886] req@ffff8dd203144700 x1876699495979904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760886.667560:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699495979904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760886.667570:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760886.667574:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760886.667577:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495979904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760886.667599:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495980032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760886.667619:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760886.667630:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760886/real 1789760886] req@ffff8dd203145500 x1876699495980032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760886.667641:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699495980032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760886.667648:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760886.667651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760886.667654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495980032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760886.667659:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495980160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760886.667673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760886.667681:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760886/real 1789760886] req@ffff8dd203144e00 x1876699495980160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760886.667690:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699495980160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760886.667703:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760886.667705:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760886.667708:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495980160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760886.667713:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495980288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760886.667725:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760886.667733:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760886/real 1789760886] req@ffff8dd203146d80 x1876699495980288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760886.667743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699495980288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760886.667749:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760886.667752:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760886.667754:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495980288:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760887.243277:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760888.267290:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760889.291326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760890.315404:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760891.339373:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760891.787229:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760891.787238:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760891.787242:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760891.787250:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 871 00000100:00080000:3.0:1789760891.787260:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760891.787267:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760891.787302:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699495980416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760891.787323:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760891.787326:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760891.787327:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760891.787330:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 871 00000100:00080000:3.0:1789760891.787333:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760891.787336:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760891.787344:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760891.787353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699495980544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760891.787359:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495980416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789760891.787364:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760891.787368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760891.787369:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760891.787374:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 871 00000100:00080000:3.0:1789760891.787376:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760891.787378:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760891.787383:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760891.787392:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699495980672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760891.787400:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760891.787402:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760891.787404:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760891.787406:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 871 00000100:00080000:3.0:1789760891.787409:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760891.787412:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760891.787414:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760891.787422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699495980800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760891.787457:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495980544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760891.787480:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495980672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760891.787508:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495980800:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760892.363352:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760893.387411:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789760893.387525:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.109:988 00000400:00000100:1.0:1789760893.387574:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760893.387590:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760893.387626:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760893.387633:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760893.387641:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789760893.387677:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760891/real 1789760893] req@ffff8dd203147b80 x1876699495980544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760893.387702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699495980544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760893.387716:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760893.387722:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760893.387727:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495980544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760893.387737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760891/real 1789760893] req@ffff8dd203146d80 x1876699495980672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760893.387751:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699495980672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760893.387760:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760893.387764:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760893.387768:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495980672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760893.387776:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760891/real 1789760893] req@ffff8dd203144e00 x1876699495980800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760893.387787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699495980800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760893.387795:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760893.387799:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760893.387803:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495980800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760893.387810:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760891/real 1789760893] req@ffff8dd203146a00 x1876699495980416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760893.387822:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699495980416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760893.387830:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760893.387834:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760893.387837:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495980416:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789760894.411381:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760895.435397:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760896.459405:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760896.907228:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760896.907238:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760896.907242:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760896.907253:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 876 00000100:00080000:3.0:1789760896.907264:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760896.907273:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760896.907310:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699495980928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760896.907338:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760896.907343:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760896.907345:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760896.907351:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 876 00000100:00080000:3.0:1789760896.907356:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760896.907360:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760896.907367:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495980928:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789760896.907371:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760896.907389:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699495981056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760896.907403:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760896.907406:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760896.907408:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760896.907413:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 876 00000100:00080000:3.0:1789760896.907417:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789760896.907424:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789760896.907425:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760896.907434:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760896.907468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699495981184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760896.907481:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760896/real 1789760896] req@ffff8dd203145500 x1876699495980928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760896.907481:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760896.907496:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760896.907500:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789760896.907501:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699495980928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760896.907512:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 876 00000100:00080000:0.0:1789760896.907515:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760896.907520:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760896.907521:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760896.907528:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495980928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789760896.907531:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760896.907539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760896.907552:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495981056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760896.907556:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699495981312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789760896.907574:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760896.907588:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760896/real 1789760896] req@ffff8dd203144700 x1876699495981056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760896.907600:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699495981056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760896.907606:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760896.907609:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760896.907611:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495981056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760896.907615:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495981184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760896.907627:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760896.907633:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760896/real 1789760896] req@ffff8dd203144a80 x1876699495981184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760896.907640:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699495981184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760896.907645:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760896.907647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760896.907649:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495981184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760896.907657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495981312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760896.907667:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760896.907674:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760896/real 1789760896] req@ffff8dd203144380 x1876699495981312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760896.907686:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699495981312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760896.907695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760896.907698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760896.907701:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495981312:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789760897.483310:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760898.507298:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760899.531284:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760900.555370:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760901.579350:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760902.027219:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760902.027229:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760902.027233:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760902.027255:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 881 00000100:00080000:3.0:1789760902.027282:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760902.027290:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760902.027369:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699495981440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760902.027398:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760902.027443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760902.027446:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760902.027451:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 881 00000100:00080000:3.0:1789760902.027455:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760902.027458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760902.027485:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760902.027504:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699495981568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760902.027520:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760902.027525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760902.027527:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760902.027556:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 881 00000100:00080000:3.0:1789760902.027562:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760902.027590:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760902.027593:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495981440:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789760902.027603:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760902.027621:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699495981696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760902.027639:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760902.027642:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760902.027643:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760902.027647:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 881 00000100:00080000:3.0:1789760902.027650:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760902.027653:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760902.027658:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760902.027668:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699495981824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789760902.027747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760902.027844:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760902/real 1789760902] req@ffff8dd203145f80 x1876699495981440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760902.027870:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699495981440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760902.027884:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760902.027889:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760902.027894:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495981440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760902.027903:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495981568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760902.027946:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760902.027958:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760902/real 1789760902] req@ffff8dd203145180 x1876699495981568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760902.027970:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699495981568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760902.027983:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760902.027986:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760902.027989:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495981568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760902.028030:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495981696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760902.028046:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760902.028055:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760902/real 1789760902] req@ffff8dd203145c00 x1876699495981696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760902.028065:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699495981696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760902.028073:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760902.028075:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760902.028088:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495981696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760902.028093:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495981824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760902.028107:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760902.028115:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760902/real 1789760902] req@ffff8dd203146300 x1876699495981824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760902.028125:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699495981824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760902.028131:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760902.028133:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760902.028137:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495981824:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760902.603390:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760903.627334:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760904.651451:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760905.675291:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760906.699322:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760907.147310:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760907.147324:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760907.147329:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760907.147343:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 886 00000100:00080000:1.0:1789760907.147386:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760907.147397:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789760907.147448:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699495981952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760907.147482:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760907.147487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760907.147489:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760907.147494:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 886 00000100:00080000:1.0:1789760907.147498:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760907.147503:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760907.147515:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760907.147532:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699495982080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760907.147543:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760907.147547:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760907.147548:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760907.147552:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 886 00000100:00080000:1.0:1789760907.147555:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760907.147560:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760907.147565:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760907.147579:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495981952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760907.147579:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699495982208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760907.147601:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760907.147604:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760907.147605:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760907.147609:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 886 00000100:00080000:1.0:1789760907.147612:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760907.147616:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760907.147621:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760907.147633:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699495982336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789760907.147681:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760907.147750:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760907/real 1789760907] req@ffff8dd20504ca80 x1876699495981952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760907.147777:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699495981952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760907.147790:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760907.147795:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760907.147799:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495981952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760907.147808:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495982080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760907.147843:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760907.147866:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760907/real 1789760907] req@ffff8dd20504dc00 x1876699495982080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760907.147887:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699495982080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760907.147901:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760907.147906:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760907.147910:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495982080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760907.147942:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495982208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760907.147968:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760907.147980:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760907/real 1789760907] req@ffff8dd20504c380 x1876699495982208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760907.147993:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699495982208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760907.148001:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760907.148005:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760907.148008:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495982208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760907.148013:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495982336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760907.148029:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760907.148037:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760907/real 1789760907] req@ffff8dd20504ea00 x1876699495982336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760907.148049:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699495982336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760907.148057:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760907.148060:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760907.148063:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495982336:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760907.723360:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760908.747346:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760909.771265:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760910.795304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760911.819331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760912.267306:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760912.267317:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760912.267321:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760912.267345:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 891 00000100:00080000:2.0:1789760912.267358:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760912.267366:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789760912.267420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699495982464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760912.267445:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760912.267448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760912.267449:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760912.267455:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 891 00000100:00080000:2.0:1789760912.267458:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760912.267461:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760912.267471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760912.267481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699495982592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760912.267489:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760912.267492:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760912.267494:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789760912.267497:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495982464:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789760912.267499:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 891 00000100:00080000:2.0:1789760912.267503:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760912.267509:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760912.267525:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760912.267534:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699495982720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760912.267542:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760912.267545:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760912.267546:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760912.267548:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 891 00000100:00080000:2.0:1789760912.267550:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760912.267553:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760912.267555:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760912.267564:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699495982848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760912.267642:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495982592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760912.267673:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495982720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760912.267690:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495982848:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760912.843284:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789760913.547630:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.109:988 00000400:00000100:1.0:1789760913.547683:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760913.547702:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760913.547747:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760913.547758:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789760913.547768:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789760913.547816:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760912/real 1789760913] req@ffff8dd205267100 x1876699495982592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760913.547840:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699495982592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760913.547851:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760913.547855:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760913.547859:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495982592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760913.547866:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760912/real 1789760913] req@ffff8dd205267800 x1876699495982720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760913.547875:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699495982720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760913.547881:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760913.547884:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760913.547886:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495982720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760913.547891:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760912/real 1789760913] req@ffff8dd205264e00 x1876699495982848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760913.547904:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699495982848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760913.547915:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760913.547919:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760913.547923:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495982848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760913.547931:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760912/real 1789760913] req@ffff8dd205267b80 x1876699495982464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760913.547946:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699495982464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760913.547956:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760913.547961:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760913.547964:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495982464:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789760913.867331:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760914.891398:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760915.915298:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760916.939405:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760917.387351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760917.387361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760917.387365:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760917.387375:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 896 00000100:00080000:3.0:1789760917.387386:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760917.387393:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760917.387429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699495982976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760917.387455:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760917.387459:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760917.387460:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760917.387464:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 896 00000100:00080000:3.0:1789760917.387468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760917.387471:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760917.387481:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760917.387493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699495983104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760917.387503:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760917.387506:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760917.387508:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760917.387511:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 896 00000100:00080000:3.0:1789760917.387514:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760917.387517:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760917.387522:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760917.387531:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495982976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789760917.387540:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699495983232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760917.387555:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760917.387559:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760917.387562:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760917.387565:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 896 00000100:00080000:3.0:1789760917.387571:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760917.387573:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760917.387578:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760917.387588:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699495983360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789760917.387630:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760917.387702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760917/real 1789760917] req@ffff8dd203147480 x1876699495982976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760917.387725:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699495982976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760917.387746:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760917.387751:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760917.387755:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495982976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760917.387763:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495983104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760917.387787:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760917.387799:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760917/real 1789760917] req@ffff8dd203144000 x1876699495983104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760917.387811:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699495983104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760917.387818:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760917.387822:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760917.387825:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495983104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760917.387856:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495983232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760917.387873:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760917.387882:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760917/real 1789760917] req@ffff8dd203147100 x1876699495983232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760917.387921:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699495983232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760917.387929:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760917.387931:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760917.387934:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495983232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760917.387940:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495983360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760917.387957:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760917.387982:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760917/real 1789760917] req@ffff8dd203147800 x1876699495983360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760917.387994:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699495983360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760917.388002:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760917.388014:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760917.388017:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495983360:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760917.963344:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760918.987362:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760920.011312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760921.035315:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760922.059389:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760922.507245:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760922.507266:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760922.507270:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760922.507278:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 902 00000100:00080000:3.0:1789760922.507287:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760922.507303:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760922.507340:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699495983488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760922.507361:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760922.507365:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760922.507367:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760922.507371:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 902 00000100:00080000:3.0:1789760922.507375:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760922.507379:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760922.507388:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760922.507402:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699495983616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760922.507408:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495983488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789760922.507413:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760922.507416:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760922.507417:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760922.507421:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 902 00000100:00080000:3.0:1789760922.507424:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760922.507427:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760922.507433:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760922.507442:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699495983744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760922.507451:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760922.507454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760922.507455:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760922.507458:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 902 00000100:00080000:3.0:1789760922.507461:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760922.507464:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789760922.507468:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1789760922.507468:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760922.507478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699495983872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760922.507506:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760922/real 1789760922] req@ffff8dd203145880 x1876699495983488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760922.507534:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699495983488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760922.507542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760922.507545:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760922.507548:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495983488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760922.507568:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495983616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760922.507587:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760922.507596:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760922/real 1789760922] req@ffff8dd203147800 x1876699495983616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760922.507604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699495983616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760922.507609:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760922.507611:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760922.507613:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495983616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760922.507616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495983744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760922.507627:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760922.507633:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760922/real 1789760922] req@ffff8dd203147100 x1876699495983744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760922.507640:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699495983744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760922.507645:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760922.507647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760922.507649:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495983744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760922.507652:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495983872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760922.507661:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760922.507667:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760922/real 1789760922] req@ffff8dd203144000 x1876699495983872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760922.507675:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699495983872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760922.507680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760922.507681:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760922.507683:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495983872:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760923.083337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760924.107357:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760925.131295:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760926.155304:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760927.179286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760927.627234:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760927.627243:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760927.627246:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760927.627255:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 907 00000100:00080000:3.0:1789760927.627267:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760927.627274:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760927.627314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699495984000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760927.627338:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760927.627341:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760927.627343:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760927.627346:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 907 00000100:00080000:3.0:1789760927.627349:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760927.627353:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760927.627361:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760927.627372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699495984128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760927.627376:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495984000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789760927.627387:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760927.627390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760927.627391:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760927.627394:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 907 00000100:00080000:3.0:1789760927.627396:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760927.627398:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760927.627401:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760927.627409:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699495984256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760927.627415:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760927.627418:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760927.627419:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760927.627421:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 907 00000100:00080000:3.0:1789760927.627425:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760927.627428:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760927.627430:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760927.627436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699495984384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789760927.627449:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760927.627494:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760927/real 1789760927] req@ffff8dd203147480 x1876699495984000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760927.627507:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699495984000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760927.627515:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760927.627518:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760927.627521:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495984000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760927.627539:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495984128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760927.627554:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760927.627562:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760927/real 1789760927] req@ffff8dd203146300 x1876699495984128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760927.627570:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699495984128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760927.627576:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760927.627578:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760927.627580:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495984128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760927.627584:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495984256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760927.627595:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760927.627600:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760927/real 1789760927] req@ffff8dd203145c00 x1876699495984256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760927.627615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699495984256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760927.627620:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760927.627622:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760927.627624:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495984256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760927.627628:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495984384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760927.627639:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760927.627645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760927/real 1789760927] req@ffff8dd203145180 x1876699495984384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760927.627652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699495984384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760927.627661:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760927.627664:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760927.627666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495984384:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760928.203317:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760929.227340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760930.251314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760931.275302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760932.299310:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760932.747376:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760932.747393:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760932.747400:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760932.747434:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 912 00000100:00080000:1.0:1789760932.747466:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760932.747476:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789760932.747531:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699495984512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760932.747562:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760932.747567:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760932.747569:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760932.747574:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 912 00000100:00080000:1.0:1789760932.747578:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760932.747582:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760932.747594:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760932.747614:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699495984640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760932.747649:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760932.747657:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789760932.747658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495984512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789760932.747665:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760932.747695:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 912 00000100:00080000:1.0:1789760932.747702:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760932.747707:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760932.747714:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760932.747740:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699495984768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760932.747752:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760932.747756:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760932.747758:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760932.747763:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 912 00000100:00080000:1.0:1789760932.747769:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789760932.747771:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789760932.747775:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789760932.747782:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760932.747796:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699495984896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760932.747838:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760932/real 1789760932] req@ffff8dd20504c000 x1876699495984512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760932.747861:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699495984512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760932.747873:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760932.747878:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760932.747882:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495984512:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760932.747909:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495984640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760932.747937:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760932.747949:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760932/real 1789760932] req@ffff8dd20504d880 x1876699495984640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760932.747959:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699495984640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760932.747965:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760932.747968:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760932.747971:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495984640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789760932.747975:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495984768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789760932.747990:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760932.747997:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760932/real 1789760932] req@ffff8dd20504ed80 x1876699495984768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760932.748005:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699495984768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760932.748011:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760932.748014:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760932.748016:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495984768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789760932.748020:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495984896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789760932.748030:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760932.748036:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760932/real 1789760932] req@ffff8dd20504f100 x1876699495984896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760932.748044:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699495984896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760932.748049:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760932.748051:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789760932.748054:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495984896:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760933.323340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760934.347323:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760935.371276:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760936.395312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760937.419282:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760937.867299:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760937.867307:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760937.867310:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760937.867318:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 917 00000100:00080000:0.0:1789760937.867328:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760937.867334:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760937.867366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699495985024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760937.867390:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760937.867393:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760937.867398:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760937.867401:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 917 00000100:00080000:0.0:1789760937.867403:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760937.867406:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760937.867414:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760937.867422:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699495985152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760937.867438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760937.867441:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495985024:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789760937.867445:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760937.867450:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760937.867453:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 917 00000100:00080000:0.0:1789760937.867455:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760937.867458:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760937.867462:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760937.867471:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699495985280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760937.867479:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760937.867481:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760937.867483:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760937.867485:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 917 00000100:00080000:0.0:1789760937.867487:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760937.867490:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760937.867493:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760937.867500:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699495985408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760937.867508:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789760937.867553:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760937/real 1789760937] req@ffff8dd211835f80 x1876699495985024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760937.867566:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699495985024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760937.867578:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760937.867581:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760937.867584:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495985024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789760937.867603:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495985152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789760937.867619:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789760937.867628:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760937/real 1789760937] req@ffff8dd211834700 x1876699495985152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760937.867636:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699495985152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760937.867641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760937.867643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760937.867645:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495985152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789760937.867649:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495985280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789760937.867660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789760937.867666:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760937/real 1789760937] req@ffff8dd211834000 x1876699495985280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760937.867673:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699495985280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760937.867678:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760937.867680:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760937.867682:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495985280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789760937.867686:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495985408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789760937.867695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789760937.867701:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760937/real 1789760937] req@ffff8dd211836a00 x1876699495985408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760937.867709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699495985408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760937.867713:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760937.867715:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789760937.867717:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495985408:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760938.443291:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760939.467320:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760940.491299:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760941.515282:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760942.539304:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760942.987248:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760942.987259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760942.987263:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760942.987275:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 922 00000100:00080000:3.0:1789760942.987288:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760942.987296:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760942.987337:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699495985536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760942.987365:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760942.987368:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760942.987370:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760942.987374:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 922 00000100:00080000:3.0:1789760942.987376:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760942.987378:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760942.987389:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760942.987398:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699495985664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760942.987406:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760942.987408:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760942.987409:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760942.987412:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 922 00000100:00080000:3.0:1789760942.987414:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760942.987419:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789760942.987420:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495985536:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789760942.987423:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760942.987434:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699495985792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760942.987441:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760942.987443:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760942.987444:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760942.987447:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 922 00000100:00080000:3.0:1789760942.987450:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760942.987452:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760942.987455:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760942.987463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699495985920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789760942.987499:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760942.987538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760942/real 1789760942] req@ffff8dd203145f80 x1876699495985536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760942.987552:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699495985536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760942.987561:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760942.987565:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760942.987568:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495985536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760942.987574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495985664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760942.987590:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760942.987599:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760942/real 1789760942] req@ffff8dd203144380 x1876699495985664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760942.987607:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699495985664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760942.987613:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760942.987615:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760942.987617:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495985664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760942.987640:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495985792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760942.987653:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760942.987659:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760942/real 1789760942] req@ffff8dd203144a80 x1876699495985792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760942.987667:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699495985792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760942.987672:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760942.987674:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760942.987675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495985792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760942.987679:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495985920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760942.987690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760942.987696:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760942/real 1789760942] req@ffff8dd203144700 x1876699495985920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760942.987704:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699495985920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760942.987710:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760942.987712:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760942.987713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495985920:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760943.563367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760944.587328:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760945.611342:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760946.635330:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760947.659308:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760948.107254:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760948.107262:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760948.107266:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760948.107274:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 927 00000100:00080000:2.0:1789760948.107283:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760948.107290:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789760948.107319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699495986048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760948.107340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760948.107343:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760948.107344:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760948.107347:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 927 00000100:00080000:2.0:1789760948.107349:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760948.107357:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760948.107366:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760948.107375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699495986176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760948.107382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760948.107385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760948.107386:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760948.107389:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 927 00000100:00080000:2.0:1789760948.107391:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760948.107394:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760948.107397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760948.107408:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699495986304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760948.107415:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495986048:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789760948.107420:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760948.107431:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760948.107433:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760948.107436:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 927 00000100:00080000:2.0:1789760948.107438:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760948.107441:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789760948.107446:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760948.107459:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699495986432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760948.107564:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495986176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760948.107610:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495986304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760948.107629:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495986432:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760948.683346:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760949.707332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760950.731332:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760951.179418:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789760951.179454:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760951.179472:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789760951.179514:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789760951.179525:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789760951.179536:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789760951.179564:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760948/real 1789760951] req@ffff8dd205264380 x1876699495986304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760951.179590:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699495986304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760951.179603:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760951.179609:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760951.179614:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495986304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760951.179623:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760948/real 1789760951] req@ffff8dd205264000 x1876699495986432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760951.179637:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699495986432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760951.179645:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760951.179650:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760951.179653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495986432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760951.179660:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760948/real 1789760951] req@ffff8dd205265880 x1876699495986048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760951.179673:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699495986048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760951.179681:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760951.179684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760951.179687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495986048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760951.179694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760948/real 1789760951] req@ffff8dd205266a00 x1876699495986176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760951.179706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699495986176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760951.179713:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760951.179717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760951.179720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495986176:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760951.755337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760952.779309:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760953.227363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760953.227373:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760953.227378:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760953.227388:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 932 00000100:00080000:0.0:1789760953.227401:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760953.227410:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760953.227455:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699495986560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760953.227488:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760953.227492:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760953.227495:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760953.227508:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 932 00000100:00080000:0.0:1789760953.227513:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760953.227519:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760953.227533:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760953.227550:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495986560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760953.227553:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699495986688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760953.227570:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760953.227573:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760953.227575:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760953.227580:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 932 00000100:00080000:0.0:1789760953.227585:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760953.227589:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760953.227596:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760953.227612:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699495986816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760953.227624:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760953.227629:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760953.227631:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760953.227635:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 932 00000100:00080000:0.0:1789760953.227641:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789760953.227641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789760953.227650:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760953.227657:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760953.227672:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699495986944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760953.227703:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760953/real 1789760953] req@ffff8dd211835c00 x1876699495986560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760953.227726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699495986560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760953.227740:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760953.227745:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760953.227750:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495986560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760953.227781:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495986688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760953.227807:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760953.227822:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760953/real 1789760953] req@ffff8dd211837800 x1876699495986688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760953.227836:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699495986688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760953.227845:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760953.227849:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760953.227853:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495986688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760953.227860:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495986816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760953.227882:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760953.227893:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760953/real 1789760953] req@ffff8dd211837b80 x1876699495986816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760953.227912:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699495986816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760953.227921:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760953.227925:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760953.227928:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495986816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760953.227934:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495986944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760953.227956:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760953.227968:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760953/real 1789760953] req@ffff8dd211837100 x1876699495986944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760953.227981:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699495986944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760953.227990:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760953.227993:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760953.227997:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495986944:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760953.803305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760954.827370:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760955.851338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760956.875289:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760957.899305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760958.347253:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760958.347262:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760958.347265:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760958.347274:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 937 00000100:00080000:3.0:1789760958.347284:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760958.347290:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760958.347334:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699495987072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760958.347363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760958.347366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760958.347368:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760958.347372:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 937 00000100:00080000:3.0:1789760958.347376:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760958.347380:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760958.347389:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760958.347399:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495987072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789760958.347411:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699495987200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760958.347425:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760958.347429:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760958.347432:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760958.347437:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 937 00000100:00080000:3.0:1789760958.347442:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760958.347447:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760958.347453:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789760958.347455:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789760958.347477:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699495987328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760958.347487:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760958.347490:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760958.347491:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760958.347495:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 937 00000100:00080000:3.0:1789760958.347498:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789760958.347502:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760958/real 1789760958] req@ffff8dd203145500 x1876699495987072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760958.347505:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760958.347512:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760958.347522:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699495987072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760958.347526:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699495987456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760958.347560:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760958.347565:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760958.347568:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495987072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760958.347589:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495987200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760958.347605:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760958.347614:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760958/real 1789760958] req@ffff8dd203146a00 x1876699495987200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760958.347622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699495987200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760958.347627:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760958.347629:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760958.347631:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495987200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760958.347635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495987328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760958.347646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760958.347652:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760958/real 1789760958] req@ffff8dd203144e00 x1876699495987328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760958.347659:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699495987328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760958.347664:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760958.347666:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760958.347668:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495987328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760958.347672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495987456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760958.347681:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760958.347687:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760958/real 1789760958] req@ffff8dd203146d80 x1876699495987456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760958.347695:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699495987456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760958.347699:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760958.347701:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760958.347703:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495987456:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760958.923372:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760959.947301:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760960.971326:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760961.995394:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760963.019358:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760963.467318:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760963.467330:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760963.467335:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760963.467347:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 943 00000100:00080000:3.0:1789760963.467359:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760963.467367:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760963.467414:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699495987584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760963.467449:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760963.467453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760963.467455:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760963.467458:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 943 00000100:00080000:3.0:1789760963.467461:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760963.467464:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760963.467490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760963.467505:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699495987712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760963.467513:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760963.467516:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760963.467518:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760963.467523:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 943 00000100:00080000:3.0:1789760963.467528:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760963.467535:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789760963.467540:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495987584:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789760963.467541:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760963.467555:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699495987840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760963.467566:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760963.467569:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760963.467570:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760963.467575:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 943 00000100:00080000:3.0:1789760963.467579:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760963.467583:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760963.467586:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760963.467596:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699495987968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789760963.467625:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760963.467665:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760963/real 1789760963] req@ffff8dd203147b80 x1876699495987584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760963.467682:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699495987584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760963.467694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760963.467699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760963.467703:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495987584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760963.467712:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495987712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760963.467735:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760963.467745:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760963/real 1789760963] req@ffff8dd203146d80 x1876699495987712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760963.467754:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699495987712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760963.467760:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760963.467763:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760963.467765:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495987712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760963.467786:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495987840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760963.467799:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760963.467806:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760963/real 1789760963] req@ffff8dd203144e00 x1876699495987840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760963.467813:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699495987840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760963.467819:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760963.467821:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760963.467823:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495987840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760963.467827:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495987968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760963.467837:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760963.467843:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760963/real 1789760963] req@ffff8dd203146a00 x1876699495987968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760963.467850:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699495987968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760963.467860:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760963.467862:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760963.467865:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495987968:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760964.043309:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760965.067329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760966.091297:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760967.115328:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760968.139357:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760968.587372:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760968.587385:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760968.587391:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760968.587405:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 948 00000100:00080000:0.0:1789760968.587419:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760968.587428:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760968.587495:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699495988096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760968.587527:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760968.587531:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760968.587533:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760968.587537:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 948 00000100:00080000:0.0:1789760968.587540:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760968.587552:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760968.587563:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760968.587577:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699495988224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760968.587587:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760968.587590:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760968.587592:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760968.587595:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 948 00000100:00080000:0.0:1789760968.587598:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760968.587602:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760968.587617:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760968.587627:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699495988352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760968.587637:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760968.587640:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760968.587642:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760968.587645:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 948 00000100:00080000:0.0:1789760968.587648:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760968.587652:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760968.587656:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760968.587667:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699495988480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760968.587677:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495988096:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1789760968.587796:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760968.587842:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760968/real 1789760968] req@ffff8dd211836300 x1876699495988096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760968.587861:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699495988096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760968.587894:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760968.587900:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760968.587905:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495988096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760968.587958:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495988224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760968.587991:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760968.588010:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760968/real 1789760968] req@ffff8dd211837100 x1876699495988224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760968.588030:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699495988224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760968.588043:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760968.588047:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760968.588050:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495988224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760968.588058:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495988352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760968.588080:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760968.588091:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760968/real 1789760968] req@ffff8dd211837b80 x1876699495988352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760968.588102:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699495988352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760968.588110:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760968.588113:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760968.588116:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495988352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760968.588142:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495988480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760968.588157:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760968.588166:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760968/real 1789760968] req@ffff8dd211837800 x1876699495988480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760968.588176:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699495988480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760968.588183:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760968.588230:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760968.588233:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495988480:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760969.163364:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760970.187347:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760971.211357:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760972.235332:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760973.259377:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760973.707280:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760973.707292:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760973.707297:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760973.707308:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 953 00000100:00080000:0.0:1789760973.707321:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760973.707329:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760973.707368:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699495988608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760973.707401:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760973.707405:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760973.707407:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760973.707411:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 953 00000100:00080000:0.0:1789760973.707414:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760973.707417:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760973.707425:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760973.707435:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699495988736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760973.707442:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760973.707444:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760973.707446:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760973.707450:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 953 00000100:00080000:0.0:1789760973.707452:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760973.707455:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760973.707460:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760973.707470:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495988608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789760973.707475:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699495988864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760973.707485:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760973.707487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760973.707489:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760973.707492:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 953 00000100:00080000:0.0:1789760973.707494:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760973.707496:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760973.707499:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760973.707507:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699495988992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789760973.707536:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760973.707571:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760973/real 1789760973] req@ffff8dd211835c00 x1876699495988608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760973.707585:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699495988608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760973.707593:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760973.707596:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760973.707599:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495988608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760973.707605:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495988736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760973.707625:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760973.707638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760973/real 1789760973] req@ffff8dd211836a00 x1876699495988736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760973.707651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699495988736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760973.707660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760973.707663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760973.707667:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495988736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760973.707691:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495988864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760973.707708:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760973.707715:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760973/real 1789760973] req@ffff8dd211834000 x1876699495988864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760973.707723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699495988864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760973.707733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760973.707735:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760973.707737:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495988864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760973.707741:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495988992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760973.707751:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760973.707756:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760973/real 1789760973] req@ffff8dd211834700 x1876699495988992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760973.707764:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699495988992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760973.707769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760973.707770:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760973.707772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495988992:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760974.283270:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760975.307362:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760976.331361:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760977.355434:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760978.379381:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760978.827329:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760978.827340:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760978.827344:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760978.827372:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 958 00000100:00080000:0.0:1789760978.827384:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760978.827391:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760978.827437:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699495989120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760978.827470:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760978.827473:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760978.827474:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760978.827478:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 958 00000100:00080000:0.0:1789760978.827480:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760978.827482:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760978.827491:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760978.827500:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699495989248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760978.827509:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760978.827512:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760978.827529:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760978.827532:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 958 00000100:00080000:0.0:1789760978.827533:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760978.827536:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760978.827540:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760978.827550:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699495989376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760978.827553:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495989120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789760978.827558:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760978.827563:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760978.827564:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760978.827567:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 958 00000100:00080000:0.0:1789760978.827570:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760978.827572:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760978.827575:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760978.827582:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699495989504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789760978.827640:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760978.827707:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760978/real 1789760978] req@ffff8dd211835f80 x1876699495989120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760978.827731:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699495989120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760978.827745:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760978.827750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760978.827756:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495989120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760978.827765:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495989248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760978.827794:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760978.827809:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760978/real 1789760978] req@ffff8dd211834380 x1876699495989248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760978.827823:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699495989248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760978.827854:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760978.827858:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760978.827863:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495989248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760978.827893:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495989376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760978.827916:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760978.827928:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760978/real 1789760978] req@ffff8dd211836680 x1876699495989376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760978.827941:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699495989376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760978.827951:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760978.827955:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760978.827959:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495989376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760978.827965:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495989504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760978.827986:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760978.827998:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760978/real 1789760978] req@ffff8dd211834a80 x1876699495989504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760978.828012:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699495989504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760978.828021:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760978.828024:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760978.828028:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495989504:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760979.403340:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760980.427374:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760981.451395:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760982.475342:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760983.499424:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760983.947294:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760983.947306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760983.947311:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760983.947323:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 963 00000100:00080000:0.0:1789760983.947335:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760983.947344:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760983.947388:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835880 x1876699495989632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760983.947421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760983.947425:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760983.947428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760983.947433:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 963 00000100:00080000:0.0:1789760983.947438:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760983.947442:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760983.947453:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760983.947469:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699495989760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760983.947485:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760983.947489:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760983.947493:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789760983.947496:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495989632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789760983.947501:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 963 00000100:00080000:0.0:1789760983.947509:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760983.947516:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760983.947522:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760983.947539:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699495989888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760983.947552:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760983.947557:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760983.947559:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760983.947563:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 963 00000100:00080000:0.0:1789760983.947568:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760983.947572:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760983.947579:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789760983.947580:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760983.947599:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699495990016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760983.947640:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760983/real 1789760983] req@ffff8dd211835880 x1876699495989632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760983.947660:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835880 x1876699495989632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760983.947670:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760983.947674:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760983.947678:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495989632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760983.947706:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495989760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760983.947727:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760983.947738:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760983/real 1789760983] req@ffff8dd211837480 x1876699495989760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760983.947749:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699495989760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760983.947762:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760983.947765:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760983.947768:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495989760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760983.947773:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495989888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760983.947786:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760983.947812:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760983/real 1789760983] req@ffff8dd211835180 x1876699495989888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760983.947824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699495989888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760983.947864:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760983.947868:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760983.947871:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495989888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760983.947877:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495990016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760983.947906:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760983.947925:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760983/real 1789760983] req@ffff8dd211836d80 x1876699495990016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760983.947944:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699495990016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760983.947957:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760983.947962:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760983.947968:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495990016:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760984.523321:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760985.547411:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760986.571361:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760987.595419:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760988.619349:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760989.067324:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760989.067335:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760989.067340:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760989.067351:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 968 00000100:00080000:3.0:1789760989.067364:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760989.067371:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789760989.067414:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699495990144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760989.067450:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760989.067455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760989.067457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760989.067462:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 968 00000100:00080000:3.0:1789760989.067467:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760989.067473:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760989.067486:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760989.067502:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699495990272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760989.067513:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495990144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789760989.067516:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760989.067537:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760989.067540:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760989.067545:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 968 00000100:00080000:3.0:1789760989.067548:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760989.067552:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760989.067557:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760989.067566:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699495990400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760989.067575:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760989.067577:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760989.067579:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760989.067582:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 968 00000100:00080000:3.0:1789760989.067585:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760989.067587:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789760989.067590:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760989.067598:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699495990528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789760989.067615:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760989.067659:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760989/real 1789760989] req@ffff8dd203145500 x1876699495990144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760989.067678:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699495990144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760989.067690:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760989.067695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760989.067700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495990144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760989.067726:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495990272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760989.067750:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760989.067776:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760989/real 1789760989] req@ffff8dd203144700 x1876699495990272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760989.067801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699495990272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760989.067810:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760989.067814:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760989.067818:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495990272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760989.067825:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495990400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760989.067846:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760989.067858:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760989/real 1789760989] req@ffff8dd203144a80 x1876699495990400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760989.067870:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699495990400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760989.067879:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760989.067883:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760989.067886:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495990400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760989.067894:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495990528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760989.067912:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760989.067924:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760989/real 1789760989] req@ffff8dd203144380 x1876699495990528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760989.067937:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699495990528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760989.067946:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760989.067950:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760989.067954:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495990528:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789760989.643392:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760990.667330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760991.691470:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760992.715341:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760993.739427:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760994.187311:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760994.187322:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760994.187327:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760994.187338:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 973 00000100:00080000:0.0:1789760994.187350:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760994.187358:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760994.187409:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699495990656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760994.187432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760994.187435:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760994.187437:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760994.187441:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 973 00000100:00080000:0.0:1789760994.187445:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760994.187449:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760994.187460:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760994.187476:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699495990784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760994.187482:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495990656:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789760994.187487:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760994.187490:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760994.187492:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760994.187496:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 973 00000100:00080000:0.0:1789760994.187500:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760994.187503:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760994.187509:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760994.187519:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699495990912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760994.187529:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760994.187532:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760994.187533:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760994.187537:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 973 00000100:00080000:0.0:1789760994.187542:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760994.187546:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760994.187552:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789760994.187555:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789760994.187562:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699495991040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760994.187604:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760994/real 1789760994] req@ffff8dd211834e00 x1876699495990656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760994.187624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699495990656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760994.187633:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760994.187636:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760994.187640:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495990656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760994.187667:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495990784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760994.187693:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760994.187708:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760994/real 1789760994] req@ffff8dd211836d80 x1876699495990784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760994.187736:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699495990784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760994.187742:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760994.187745:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760994.187747:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495990784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760994.187752:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495990912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760994.187769:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760994.187777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760994/real 1789760994] req@ffff8dd211835180 x1876699495990912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760994.187785:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699495990912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760994.187790:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760994.187792:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760994.187794:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495990912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760994.187798:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495991040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760994.187809:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760994.187816:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760994/real 1789760994] req@ffff8dd211837480 x1876699495991040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760994.187824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699495991040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760994.187829:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760994.187830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760994.187832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495991040:192.168.203.109@tcp:38:kworker.0 00010000:00010000:3.0:1789760994.763298:0:11116:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1789760994.763328:0:5596:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1789760994.763341:0:5596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd21299e200/0x37c40db7b5accb5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x493674835453c7a2 expref: -99 pid: 11747 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789760994.763387:0:5596:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1789760994.763410:0:5596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd21299e800/0x37c40db7b5accae lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x493674835453c786 expref: -99 pid: 11747 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789760994.763413:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760994.763422:0:5596:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd205cc3000/0x37c40db7b5acc99 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x493674835453c724 expref: -99 pid: 11740 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789760994.763487:0:5596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd21299e200/0x37c40db7b5accb5 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x493674835453c7a2 expref: -99 pid: 11747 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760994.763493:0:5596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd21299e800/0x37c40db7b5accae lrc: 1/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x493674835453c786 expref: -99 pid: 11747 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760994.763497:0:5596:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd205cc3000/0x37c40db7b5acc99 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x493674835453c724 expref: -99 pid: 11740 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789760994.763501:0:5596:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1789760994.763528:0:5596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd21299e200/0x37c40db7b5accb5 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x493674835453c7a2 expref: -99 pid: 11747 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760994.763561:0:5596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd21299e800/0x37c40db7b5accae lrc: 0/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x493674835453c786 expref: -99 pid: 11747 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760994.763567:0:5596:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8dd2055fc800 lock: ffff8dd205cc3000/0x37c40db7b5acc99 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x493674835453c724 expref: -99 pid: 11740 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789760994.763579:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760994.763580:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760994.763582:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760994.763615:0:2356:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8dd211835880 x1876699495991168/t0(0) o103->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:17/18 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1789760995.787317:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760996.811350:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760997.835334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760998.859355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760999.307252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760999.307264:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760999.307268:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760999.307279:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 978 00000100:00080000:0.0:1789760999.307291:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760999.307299:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789760999.307336:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699495991296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760999.307359:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760999.307362:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760999.307363:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760999.307367:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 978 00000100:00080000:0.0:1789760999.307369:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760999.307371:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760999.307379:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760999.307389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699495991424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760999.307398:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760999.307400:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760999.307401:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760999.307405:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 978 00000100:00080000:0.0:1789760999.307409:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789760999.307410:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495991296:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789760999.307414:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789760999.307421:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760999.307443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699495991552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760999.307456:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760999.307461:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760999.307463:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760999.307466:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 978 00000100:00080000:0.0:1789760999.307470:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760999.307475:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789760999.307477:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789760999.307480:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760999.307492:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699495991680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760999.307522:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760999/real 1789760999] req@ffff8dd211834a80 x1876699495991296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760999.307537:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699495991296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760999.307545:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760999.307548:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760999.307551:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495991296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789760999.307571:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495991424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760999.307589:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760999.307597:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760999/real 1789760999] req@ffff8dd211836680 x1876699495991424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760999.307605:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699495991424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760999.307611:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760999.307613:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760999.307615:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495991424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789760999.307619:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495991552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789760999.307630:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760999.307636:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760999/real 1789760999] req@ffff8dd211834380 x1876699495991552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760999.307644:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699495991552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760999.307649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760999.307651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760999.307653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495991552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789760999.307656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495991680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789760999.307666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789760999.307672:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760999/real 1789760999] req@ffff8dd211835f80 x1876699495991680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760999.307699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699495991680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760999.307705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760999.307707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789760999.307709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495991680:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789760999.883313:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761000.907329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761001.931309:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761002.955392:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761003.979337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761004.427362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761004.427375:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761004.427381:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761004.427395:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 984 00000100:00080000:0.0:1789761004.427411:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761004.427422:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761004.427471:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699495991808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761004.427530:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761004.427536:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761004.427538:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761004.427544:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 984 00000100:00080000:0.0:1789761004.427550:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761004.427555:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761004.427569:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761004.427608:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699495991936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761004.427621:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495991808:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761004.427634:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761004.427658:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761004.427662:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761004.427699:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 984 00000100:00080000:0.0:1789761004.427705:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761004.427712:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761004.427721:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761004.427742:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699495992064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761004.427758:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761004.427762:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761004.427767:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761004.427774:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761004.427779:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 984 00000100:00080000:0.0:1789761004.427788:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761004.427801:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761004.427808:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761004.427824:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699495992192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761004.427896:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761004/real 1789761004] req@ffff8dd211834700 x1876699495991808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761004.427937:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699495991808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761004.427967:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761004.427975:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761004.427983:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495991808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761004.428025:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495991936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761004.428082:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761004.428104:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761004/real 1789761004] req@ffff8dd211834000 x1876699495991936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761004.428160:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699495991936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761004.428227:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761004.428232:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761004.428237:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495991936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761004.428246:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495992064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761004.428294:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761004.428307:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761004/real 1789761004] req@ffff8dd211836a00 x1876699495992064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761004.428318:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699495992064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761004.428326:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761004.428329:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761004.428332:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495992064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761004.428337:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495992192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761004.428353:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761004.428362:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761004/real 1789761004] req@ffff8dd211835c00 x1876699495992192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761004.428372:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699495992192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761004.428380:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761004.428382:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761004.428386:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495992192:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761005.003359:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761006.027341:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761007.051357:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761008.075358:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761009.099309:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761009.547288:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761009.547300:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761009.547305:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761009.547316:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 989 00000100:00080000:0.0:1789761009.547328:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761009.547336:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761009.547381:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699495992320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761009.547407:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761009.547410:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761009.547412:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761009.547415:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 989 00000100:00080000:0.0:1789761009.547418:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761009.547420:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761009.547429:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761009.547441:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699495992448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761009.547449:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761009.547452:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761009.547453:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761009.547458:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 989 00000100:00080000:0.0:1789761009.547460:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761009.547460:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495992320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761009.547467:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761009.547473:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761009.547480:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699495992576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761009.547492:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761009.547495:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761009.547496:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761009.547498:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 989 00000100:00080000:0.0:1789761009.547501:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761009.547503:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761009.547506:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761009.547514:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699495992704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761009.547537:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761009.547582:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761009/real 1789761009] req@ffff8dd211837800 x1876699495992320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761009.547598:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699495992320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761009.547625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761009.547628:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761009.547632:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495992320:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761009.547655:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495992448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761009.547675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761009.547685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761009/real 1789761009] req@ffff8dd211837b80 x1876699495992448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761009.547696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699495992448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761009.547705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761009.547709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761009.547713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495992448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761009.547720:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495992576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761009.547746:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761009.547756:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761009/real 1789761009] req@ffff8dd211837100 x1876699495992576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761009.547764:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699495992576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761009.547770:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761009.547773:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761009.547775:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495992576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761009.547780:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495992704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761009.547793:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761009.547799:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761009/real 1789761009] req@ffff8dd211836300 x1876699495992704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761009.547807:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699495992704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761009.547812:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761009.547814:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761009.547816:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495992704:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761010.123318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761011.147318:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761012.171293:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761013.195354:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761014.219345:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761014.667264:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761014.667272:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761014.667275:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761014.667283:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 994 00000100:00080000:2.0:1789761014.667294:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761014.667301:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761014.667338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699495992832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761014.667365:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761014.667368:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761014.667369:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761014.667373:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 994 00000100:00080000:2.0:1789761014.667376:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761014.667380:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761014.667387:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761014.667405:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699495992960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761014.667416:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495992832:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761014.667421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761014.667425:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761014.667427:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761014.667431:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 994 00000100:00080000:2.0:1789761014.667433:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761014.667438:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761014.667442:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761014.667450:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699495993088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761014.667457:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761014.667460:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761014.667461:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761014.667465:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 994 00000100:00080000:2.0:1789761014.667467:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761014.667472:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761014.667478:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761014.667490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699495993216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761014.667543:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495992960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761014.667587:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495993088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761014.667608:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495993216:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761015.243381:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761016.267333:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761017.291355:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761017.739495:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789761017.739566:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761017.739585:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761017.739624:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761017.739635:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761017.739650:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789761017.739683:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761014/real 1789761017] req@ffff8dd205264700 x1876699495992960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761017.739713:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699495992960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761017.739728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761017.739734:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761017.739738:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495992960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761017.739747:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761014/real 1789761017] req@ffff8dd205266d80 x1876699495993088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761017.739762:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699495993088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761017.739770:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761017.739774:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761017.739777:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495993088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761017.739784:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761014/real 1789761017] req@ffff8dd205265f80 x1876699495993216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761017.739796:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699495993216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761017.739805:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761017.739809:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761017.739812:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495993216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761017.739820:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761014/real 1789761017] req@ffff8dd205266300 x1876699495992832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761017.739833:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699495992832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761017.739842:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761017.739846:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761017.739850:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495992832:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789761018.315357:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761019.339294:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761019.787239:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761019.787250:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761019.787255:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761019.787266:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 999 00000100:00080000:0.0:1789761019.787278:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761019.787286:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761019.787320:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699495993344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761019.787349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761019.787353:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761019.787355:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761019.787362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 999 00000100:00080000:0.0:1789761019.787365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761019.787370:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761019.787380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761019.787392:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495993344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761019.787394:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699495993472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761019.787409:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761019.787413:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761019.787415:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761019.787420:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 999 00000100:00080000:0.0:1789761019.787424:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761019.787429:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761019.787435:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789761019.787450:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761019.787457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699495993600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761019.787470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761019.787473:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761019.787476:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761019.787480:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 999 00000100:00100000:1.0:1789761019.787485:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761019/real 1789761019] req@ffff8dd211836300 x1876699495993344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761019.787487:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761019.787496:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761019.787504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761019.787508:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699495993344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761019.787520:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699495993728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761019.787523:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761019.787551:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761019.787558:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495993344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761019.787585:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495993472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761019.787612:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761019.787625:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761019/real 1789761019] req@ffff8dd211837100 x1876699495993472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761019.787643:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699495993472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761019.787652:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761019.787655:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761019.787658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495993472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761019.787664:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495993600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761019.787687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761019.787699:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761019/real 1789761019] req@ffff8dd211837b80 x1876699495993600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761019.787710:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699495993600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761019.787718:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761019.787721:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761019.787724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495993600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761019.787730:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495993728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761019.787748:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761019.787765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761019/real 1789761019] req@ffff8dd211837800 x1876699495993728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761019.787775:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699495993728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761019.787789:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761019.787792:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761019.787795:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495993728:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789761020.363278:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761021.387380:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761022.411386:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761023.435312:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761024.459341:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761024.907218:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761024.907227:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761024.907235:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761024.907245:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1004 00000100:00080000:0.0:1789761024.907254:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761024.907260:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761024.907289:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699495993856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761024.907319:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761024.907322:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761024.907323:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761024.907326:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1004 00000100:00080000:0.0:1789761024.907328:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761024.907331:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761024.907337:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761024.907348:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699495993984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761024.907357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761024.907360:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761024.907363:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761024.907369:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1004 00000100:00100000:1.0:1789761024.907371:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495993856:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761024.907373:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761024.907381:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761024.907388:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761024.907403:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699495994112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761024.907416:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761024.907420:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761024.907422:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761024.907426:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1004 00000100:00080000:0.0:1789761024.907430:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761024.907433:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761024.907438:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789761024.907443:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761024.907458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699495994240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761024.907485:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761024/real 1789761024] req@ffff8dd211835c00 x1876699495993856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761024.907521:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699495993856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761024.907529:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761024.907532:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761024.907535:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495993856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761024.907541:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495993984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761024.907560:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761024.907569:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761024/real 1789761024] req@ffff8dd211836a00 x1876699495993984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761024.907578:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699495993984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761024.907584:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761024.907586:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761024.907589:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495993984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761024.907608:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495994112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761024.907621:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761024.907628:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761024/real 1789761024] req@ffff8dd211834000 x1876699495994112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761024.907635:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699495994112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761024.907641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761024.907643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761024.907645:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495994112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761024.907648:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495994240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761024.907658:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761024.907664:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761024/real 1789761024] req@ffff8dd211834700 x1876699495994240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761024.907672:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699495994240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761024.907677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761024.907678:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761024.907680:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495994240:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761025.483366:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761026.507360:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761027.531367:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761028.555325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761029.579325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761030.027294:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761030.027306:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761030.027311:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761030.027323:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1009 00000100:00080000:2.0:1789761030.027336:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761030.027342:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761030.027386:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699495994368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761030.027414:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761030.027417:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761030.027418:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761030.027421:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1009 00000100:00080000:2.0:1789761030.027424:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761030.027426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761030.027435:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761030.027447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699495994496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761030.027473:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761030.027476:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789761030.027484:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495994368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761030.027486:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761030.027495:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1009 00000100:00080000:2.0:1789761030.027506:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761030.027512:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761030.027518:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761030.027530:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699495994624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761030.027538:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761030.027541:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761030.027543:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761030.027546:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1009 00000100:00080000:2.0:1789761030.027550:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761030.027553:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761030.027557:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761030.027564:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699495994752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761030.027581:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761030.027630:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761030/real 1789761030] req@ffff8dd205265180 x1876699495994368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761030.027651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699495994368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761030.027663:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761030.027668:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761030.027673:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495994368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761030.027698:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495994496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761030.027723:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761030.027737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761030/real 1789761030] req@ffff8dd205266300 x1876699495994496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761030.027751:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699495994496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761030.027761:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761030.027765:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761030.027769:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495994496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761030.027776:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495994624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761030.027794:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761030.027805:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761030/real 1789761030] req@ffff8dd205265f80 x1876699495994624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761030.027818:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699495994624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761030.027834:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761030.027837:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761030.027841:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495994624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761030.027848:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495994752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761030.027865:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761030.027886:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761030/real 1789761030] req@ffff8dd205266d80 x1876699495994752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761030.027899:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699495994752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761030.027909:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761030.027912:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761030.027916:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495994752:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761030.603322:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761031.627312:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761032.651310:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761033.675347:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761034.699381:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761035.147307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761035.147319:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761035.147340:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761035.147352:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1014 00000100:00080000:2.0:1789761035.147379:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761035.147390:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761035.147458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699495994880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761035.147518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761035.147523:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761035.147526:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761035.147530:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1014 00000100:00080000:2.0:1789761035.147536:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761035.147542:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761035.147566:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761035.147574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495994880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761035.147585:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699495995008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761035.147599:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761035.147603:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761035.147605:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761035.147610:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1014 00000100:00080000:2.0:1789761035.147616:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761035.147621:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761035.147627:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761035.147643:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699495995136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761035.147646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761035.147661:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761035.147665:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761035.147667:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761035.147672:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1014 00000100:00080000:2.0:1789761035.147676:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761035.147680:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761035.147686:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761035.147695:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761035/real 1789761035] req@ffff8dd205264700 x1876699495994880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761035.147701:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699495995264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761035.147722:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699495994880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761035.147734:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761035.147740:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761035.147745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495994880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761035.147774:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495995008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761035.147800:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761035.147812:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761035/real 1789761035] req@ffff8dd205266a00 x1876699495995008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761035.147822:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699495995008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761035.147829:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761035.147831:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761035.147843:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495995008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761035.147851:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495995136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761035.147868:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761035.147876:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761035/real 1789761035] req@ffff8dd205265880 x1876699495995136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761035.147885:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699495995136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761035.147891:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761035.147894:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761035.147896:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495995136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761035.147900:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495995264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761035.147911:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761035.147917:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761035/real 1789761035] req@ffff8dd205264000 x1876699495995264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761035.147926:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699495995264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761035.147931:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761035.147933:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761035.147935:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495995264:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761035.723376:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761036.747323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761037.771316:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761038.795337:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761039.819332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761040.267310:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761040.267321:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761040.267325:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761040.267337:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1019 00000100:00080000:2.0:1789761040.267350:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761040.267358:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761040.267414:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699495995392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761040.267456:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761040.267460:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761040.267462:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761040.267467:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1019 00000100:00080000:2.0:1789761040.267472:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761040.267477:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761040.267490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761040.267517:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699495995520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761040.267525:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761040.267529:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761040.267531:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761040.267536:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1019 00000100:00080000:2.0:1789761040.267538:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761040.267541:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761040.267546:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761040.267553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699495995648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761040.267563:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761040.267566:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761040.267568:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761040.267574:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1019 00000100:00100000:1.0:1789761040.267578:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495995392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761040.267578:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761040.267590:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761040.267601:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761040.267610:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699495995776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761040.267685:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761040.267747:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761040/real 1789761040] req@ffff8dd205264380 x1876699495995392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761040.267765:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699495995392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761040.267796:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761040.267801:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761040.267806:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495995392:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761040.267815:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495995520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761040.267842:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761040.267855:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761040/real 1789761040] req@ffff8dd205267b80 x1876699495995520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761040.267865:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699495995520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761040.267895:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761040.267899:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761040.267904:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495995520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761040.267935:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495995648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761040.267976:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761040.267989:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761040/real 1789761040] req@ffff8dd205264e00 x1876699495995648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761040.268003:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699495995648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761040.268012:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761040.268016:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761040.268020:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495995648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761040.268027:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495995776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761040.268043:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761040.268051:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761040/real 1789761040] req@ffff8dd205267800 x1876699495995776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761040.268059:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699495995776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761040.268064:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761040.268066:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761040.268069:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495995776:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761040.843398:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761041.867405:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761042.891332:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761043.915365:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761044.939365:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761045.387308:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761045.387320:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761045.387324:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761045.387353:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1024 00000100:00080000:0.0:1789761045.387366:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761045.387374:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761045.387419:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699495995904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761045.387449:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761045.387452:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761045.387454:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761045.387460:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1024 00000100:00080000:0.0:1789761045.387463:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761045.387465:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761045.387475:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761045.387484:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699495996032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761045.387492:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761045.387495:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761045.387496:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761045.387502:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1024 00000100:00100000:1.0:1789761045.387502:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495995904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761045.387506:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761045.387512:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761045.387517:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761045.387525:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699495996160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761045.387533:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761045.387536:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761045.387537:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761045.387540:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1024 00000100:00080000:0.0:1789761045.387543:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761045.387546:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761045.387548:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761045.387556:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699495996288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761045.387579:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761045.387622:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761045/real 1789761045] req@ffff8dd211835f80 x1876699495995904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761045.387638:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699495995904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761045.387647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761045.387651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761045.387654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495995904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761045.387678:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495996032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761045.387693:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761045.387701:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761045/real 1789761045] req@ffff8dd211834380 x1876699495996032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761045.387709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699495996032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761045.387715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761045.387717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761045.387719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495996032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761045.387723:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495996160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761045.387737:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761045.387743:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761045/real 1789761045] req@ffff8dd211836680 x1876699495996160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761045.387751:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699495996160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761045.387757:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761045.387758:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761045.387760:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495996160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761045.387763:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495996288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761045.387773:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761045.387780:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761045/real 1789761045] req@ffff8dd211834a80 x1876699495996288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761045.387787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699495996288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761045.387792:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761045.387794:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761045.387804:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495996288:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761045.963395:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761046.987316:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761048.011358:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761049.035335:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761050.059372:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761050.507278:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761050.507288:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761050.507293:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761050.507313:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1030 00000100:00080000:0.0:1789761050.507325:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761050.507332:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761050.507369:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699495996416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761050.507395:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761050.507398:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761050.507400:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761050.507403:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1030 00000100:00080000:0.0:1789761050.507405:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761050.507408:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761050.507415:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761050.507427:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699495996544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761050.507435:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761050.507437:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761050.507438:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761050.507443:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1030 00000100:00080000:0.0:1789761050.507446:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761050.507446:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495996416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761050.507450:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761050.507457:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761050.507464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699495996672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761050.507470:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761050.507473:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761050.507474:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761050.507476:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1030 00000100:00080000:0.0:1789761050.507478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761050.507480:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761050.507483:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761050.507490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699495996800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761050.507520:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761050.507560:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761050/real 1789761050] req@ffff8dd211837480 x1876699495996416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761050.507604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699495996416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761050.507616:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761050.507621:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761050.507625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495996416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761050.507653:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495996544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761050.507678:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761050.507692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761050/real 1789761050] req@ffff8dd211835180 x1876699495996544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761050.507706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699495996544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761050.507716:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761050.507720:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761050.507739:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495996544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761050.507745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495996672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761050.507762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761050.507770:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761050/real 1789761050] req@ffff8dd211836d80 x1876699495996672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761050.507807:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699495996672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761050.507823:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761050.507827:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761050.507830:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495996672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761050.507837:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495996800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761050.507859:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761050.507871:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761050/real 1789761050] req@ffff8dd211834e00 x1876699495996800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761050.507883:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699495996800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761050.507892:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761050.507896:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761050.507899:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495996800:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761051.083354:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761052.107360:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761053.131314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761054.155390:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761055.179340:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761055.627331:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761055.627339:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761055.627343:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761055.627353:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1035 00000100:00080000:2.0:1789761055.627365:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761055.627371:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761055.627411:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699495996928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761055.627448:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761055.627451:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761055.627452:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761055.627456:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1035 00000100:00080000:2.0:1789761055.627459:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761055.627461:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761055.627470:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761055.627484:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699495997056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761055.627496:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761055.627499:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761055.627500:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761055.627503:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1035 00000100:00080000:2.0:1789761055.627506:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761055.627512:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761055.627521:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495996928:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789761055.627521:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761055.627537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699495997184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761055.627547:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761055.627550:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761055.627555:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761055.627559:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1035 00000100:00080000:2.0:1789761055.627561:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761055.627564:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761055.627567:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761055.627579:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699495997312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761055.627624:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761055.627685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761055/real 1789761055] req@ffff8dd205267100 x1876699495996928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761055.627706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699495996928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761055.627720:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761055.627726:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761055.627731:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495996928:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761055.627741:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495997056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761055.627768:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761055.627783:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761055/real 1789761055] req@ffff8dd205266680 x1876699495997056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761055.627798:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699495997056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761055.627808:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761055.627813:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761055.627817:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495997056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761055.627850:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495997184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761055.627874:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761055.627887:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761055/real 1789761055] req@ffff8dd205265500 x1876699495997184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761055.627900:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699495997184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761055.627910:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761055.627913:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761055.627917:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495997184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761055.627925:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495997312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761055.627947:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761055.627961:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761055/real 1789761055] req@ffff8dd205264a80 x1876699495997312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761055.627982:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699495997312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761055.627991:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761055.627994:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761055.627998:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495997312:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761056.203332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761057.227358:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761058.251345:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761059.275335:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761060.299305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761060.747307:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761060.747316:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761060.747319:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761060.747328:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1040 00000100:00080000:2.0:1789761060.747337:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761060.747343:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761060.747375:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699495997440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761060.747398:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761060.747402:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761060.747403:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761060.747406:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1040 00000100:00080000:2.0:1789761060.747408:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761060.747410:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761060.747418:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761060.747428:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699495997568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761060.747436:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761060.747438:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761060.747439:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761060.747442:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1040 00000100:00080000:2.0:1789761060.747443:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761060.747448:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761060.747451:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495997440:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789761060.747452:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761060.747463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699495997696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761060.747471:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761060.747473:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761060.747474:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761060.747477:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1040 00000100:00080000:2.0:1789761060.747478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761060.747481:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761060.747485:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761060.747492:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699495997824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761060.747515:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761060.747560:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761060/real 1789761060] req@ffff8dd205267480 x1876699495997440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761060.747576:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699495997440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761060.747585:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761060.747588:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761060.747591:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495997440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761060.747597:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495997568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761060.747615:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761060.747624:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761060/real 1789761060] req@ffff8dd205264a80 x1876699495997568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761060.747632:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699495997568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761060.747642:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761060.747645:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761060.747647:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495997568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761060.747665:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495997696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761060.747677:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761060.747684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761060/real 1789761060] req@ffff8dd205265500 x1876699495997696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761060.747691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699495997696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761060.747697:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761060.747698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761060.747700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495997696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761060.747704:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495997824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761060.747713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761060.747719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761060/real 1789761060] req@ffff8dd205266680 x1876699495997824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761060.747727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699495997824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761060.747731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761060.747733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761060.747736:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495997824:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761061.323276:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761062.347315:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761063.371283:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761064.395388:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761065.419342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761065.867332:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761065.867340:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761065.867344:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761065.867353:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1045 00000100:00080000:0.0:1789761065.867363:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761065.867369:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761065.867404:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699495997952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761065.867425:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761065.867428:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761065.867430:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761065.867433:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1045 00000100:00080000:0.0:1789761065.867435:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761065.867437:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761065.867444:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761065.867453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699495998080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761065.867461:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761065.867463:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761065.867464:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761065.867467:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1045 00000100:00080000:0.0:1789761065.867469:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761065.867473:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495997952:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761065.867474:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761065.867482:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761065.867491:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699495998208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761065.867498:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761065.867500:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761065.867502:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761065.867504:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1045 00000100:00080000:0.0:1789761065.867506:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761065.867508:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761065.867510:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761065.867518:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699495998336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761065.867546:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761065.867580:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761065/real 1789761065] req@ffff8dd211834e00 x1876699495997952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761065.867594:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699495997952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761065.867601:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761065.867604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761065.867608:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495997952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761065.867613:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495998080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761065.867639:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761065.867647:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761065/real 1789761065] req@ffff8dd211836d80 x1876699495998080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761065.867655:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699495998080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761065.867660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761065.867662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761065.867665:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495998080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761065.867685:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495998208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761065.867697:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761065.867704:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761065/real 1789761065] req@ffff8dd211835180 x1876699495998208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761065.867711:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699495998208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761065.867717:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761065.867718:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761065.867720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495998208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761065.867723:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495998336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761065.867743:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761065.867749:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761065/real 1789761065] req@ffff8dd211837480 x1876699495998336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761065.867764:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699495998336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761065.867769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761065.867771:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761065.867773:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495998336:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761066.443295:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761067.467348:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761068.491341:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761069.515394:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761070.539298:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761070.987264:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761070.987275:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761070.987279:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761070.987289:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1050 00000100:00080000:0.0:1789761070.987300:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761070.987308:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761070.987344:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699495998464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761070.987373:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761070.987377:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761070.987379:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761070.987385:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1050 00000100:00080000:0.0:1789761070.987391:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761070.987396:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761070.987406:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761070.987424:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495998464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761070.987425:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699495998592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761070.987451:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761070.987455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761070.987457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761070.987464:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1050 00000100:00080000:0.0:1789761070.987468:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761070.987472:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761070.987478:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761070.987494:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699495998720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761070.987506:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761070.987513:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761070.987514:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761070.987519:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761070.987525:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1050 00000100:00080000:0.0:1789761070.987531:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761070.987544:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761070.987552:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761070.987564:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761070/real 1789761070] req@ffff8dd211834a80 x1876699495998464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761070.987570:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699495998848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761070.987589:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699495998464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761070.987630:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761070.987636:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761070.987640:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495998464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761070.987671:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495998592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761070.987690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761070.987699:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761070/real 1789761070] req@ffff8dd211836680 x1876699495998592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761070.987708:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699495998592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761070.987714:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761070.987717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761070.987719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495998592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761070.987723:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495998720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761070.987735:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761070.987741:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761070/real 1789761070] req@ffff8dd211834380 x1876699495998720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761070.987748:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699495998720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761070.987754:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761070.987756:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761070.987758:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495998720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761070.987762:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495998848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761070.987773:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761070.987779:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761070/real 1789761070] req@ffff8dd211835f80 x1876699495998848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761070.987786:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699495998848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761070.987791:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761070.987793:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761070.987795:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495998848:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761071.563380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761072.587341:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761073.611325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761074.635280:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761075.659356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761076.107272:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761076.107283:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761076.107289:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761076.107300:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1055 00000100:00080000:0.0:1789761076.107311:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761076.107319:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761076.107358:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699495998976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761076.107392:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761076.107396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761076.107398:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761076.107403:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1055 00000100:00080000:0.0:1789761076.107407:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761076.107411:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761076.107420:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761076.107435:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699495999104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761076.107445:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761076.107449:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761076.107451:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761076.107458:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1055 00000100:00100000:1.0:1789761076.107459:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495998976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761076.107461:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761076.107471:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761076.107476:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761076.107489:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699495999232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761076.107500:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761076.107504:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761076.107506:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761076.107510:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1055 00000100:00080000:0.0:1789761076.107528:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761076.107534:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761076.107535:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761076.107544:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761076.107559:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699495999360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761076.107581:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761076/real 1789761076] req@ffff8dd211834700 x1876699495998976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761076.107604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699495998976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761076.107617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761076.107623:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761076.107627:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495998976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761076.107643:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495999104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761076.107663:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761076.107673:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761076/real 1789761076] req@ffff8dd211834000 x1876699495999104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761076.107682:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699495999104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761076.107687:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761076.107690:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761076.107698:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495999104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761076.107719:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495999232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761076.107732:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761076.107739:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761076/real 1789761076] req@ffff8dd211836a00 x1876699495999232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761076.107748:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699495999232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761076.107756:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761076.107759:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761076.107762:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495999232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761076.107768:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495999360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761076.107788:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761076.107800:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761076/real 1789761076] req@ffff8dd211835c00 x1876699495999360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761076.107812:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699495999360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761076.107823:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761076.107825:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761076.107827:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495999360:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761076.683348:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761077.707337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761078.731383:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761079.755311:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761080.779327:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761081.227266:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761081.227275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761081.227279:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761081.227288:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1060 00000100:00080000:0.0:1789761081.227299:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761081.227308:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761081.227348:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699495999488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761081.227376:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761081.227380:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761081.227383:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761081.227387:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1060 00000100:00080000:0.0:1789761081.227392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761081.227397:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761081.227408:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761081.227424:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699495999616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761081.227426:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495999488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761081.227438:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761081.227442:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761081.227443:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761081.227446:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1060 00000100:00080000:0.0:1789761081.227449:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761081.227451:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761081.227456:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761081.227464:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699495999744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761081.227471:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761081.227473:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761081.227474:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761081.227477:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1060 00000100:00080000:0.0:1789761081.227479:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761081.227481:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761081.227484:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761081.227491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699495999872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761081.227562:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495999616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761081.227599:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495999744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761081.227627:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495999872:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761081.803395:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761082.827378:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761083.851305:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761084.299486:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789761084.299546:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761084.299568:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761084.299613:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761084.299626:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761084.299642:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789761084.299659:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761081/real 1789761084] req@ffff8dd211837b80 x1876699495999616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761084.299691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699495999616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761084.299705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761084.299711:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761084.299716:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495999616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761084.299725:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761081/real 1789761084] req@ffff8dd211837100 x1876699495999744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761084.299739:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699495999744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761084.299748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761084.299752:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761084.299756:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699495999744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761084.299764:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761081/real 1789761084] req@ffff8dd211836300 x1876699495999872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761084.299778:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699495999872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761084.299788:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761084.299791:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761084.299794:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699495999872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761084.299802:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761081/real 1789761084] req@ffff8dd211837800 x1876699495999488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761084.299815:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699495999488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761084.299823:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761084.299826:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761084.299829:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699495999488:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789761084.875354:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761085.899306:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761086.347266:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761086.347275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761086.347279:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761086.347289:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1065 00000100:00080000:0.0:1789761086.347299:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761086.347306:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761086.347343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496000000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761086.347376:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761086.347380:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761086.347382:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761086.347386:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1065 00000100:00080000:0.0:1789761086.347390:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761086.347395:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761086.347406:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761086.347422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496000128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761086.347428:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496000000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761086.347434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761086.347438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761086.347440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761086.347444:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1065 00000100:00080000:0.0:1789761086.347447:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761086.347450:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761086.347455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761086.347465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496000256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761086.347474:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761086.347477:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761086.347478:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761086.347481:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1065 00000100:00080000:0.0:1789761086.347498:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761086.347503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761086.347508:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761086.347510:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761086.347525:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496000384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761086.347559:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761086/real 1789761086] req@ffff8dd211837800 x1876699496000000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761086.347581:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496000000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761086.347592:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761086.347597:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761086.347601:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496000000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761086.347626:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496000128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761086.347651:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761086.347665:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761086/real 1789761086] req@ffff8dd211836300 x1876699496000128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761086.347685:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496000128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761086.347695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761086.347699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761086.347702:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496000128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761086.347709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496000256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761086.347731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761086.347749:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761086/real 1789761086] req@ffff8dd211837100 x1876699496000256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761086.347763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496000256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761086.347772:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761086.347775:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761086.347779:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496000256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761086.347786:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496000384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761086.347828:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761086.347840:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761086/real 1789761086] req@ffff8dd211837b80 x1876699496000384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761086.347854:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496000384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761086.347862:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761086.347865:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761086.347869:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496000384:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761086.923374:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761087.947334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761088.971317:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761089.995302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761091.019348:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761091.467260:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761091.467271:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761091.467274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761091.467286:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1071 00000100:00080000:2.0:1789761091.467298:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761091.467307:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761091.467344:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496000512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761091.467371:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761091.467376:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761091.467378:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761091.467382:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1071 00000100:00080000:2.0:1789761091.467387:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761091.467391:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761091.467404:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761091.467420:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496000640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761091.467429:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761091.467432:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761091.467434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761091.467437:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1071 00000100:00080000:2.0:1789761091.467441:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761091.467448:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761091.467451:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496000512:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789761091.467455:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761091.467477:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496000768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761091.467492:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761091.467497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761091.467500:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761091.467512:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1071 00000100:00080000:2.0:1789761091.467517:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761091.467524:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761091.467525:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789761091.467536:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761091.467549:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496000896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761091.467587:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761091/real 1789761091] req@ffff8dd205267100 x1876699496000512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761091.467610:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496000512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761091.467623:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761091.467627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761091.467630:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496000512:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761091.467641:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496000640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761091.467668:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761091.467683:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761091/real 1789761091] req@ffff8dd205267800 x1876699496000640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761091.467696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496000640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761091.467703:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761091.467705:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761091.467707:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496000640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761091.467730:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496000768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761091.467754:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761091.467763:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761091/real 1789761091] req@ffff8dd205264e00 x1876699496000768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761091.467771:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496000768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761091.467777:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761091.467779:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761091.467782:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496000768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761091.467786:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496000896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761091.467798:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761091.467804:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761091/real 1789761091] req@ffff8dd205267b80 x1876699496000896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761091.467812:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496000896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761091.467822:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761091.467824:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761091.467826:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496000896:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761092.043330:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761093.067357:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761094.091332:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761095.115289:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761096.139328:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761096.587263:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761096.587270:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761096.587274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761096.587282:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1076 00000100:00080000:2.0:1789761096.587292:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761096.587298:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761096.587342:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496001024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761096.587362:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761096.587364:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761096.587366:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761096.587381:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1076 00000100:00080000:2.0:1789761096.587383:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761096.587385:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761096.587393:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761096.587402:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496001152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761096.587409:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761096.587412:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761096.587413:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761096.587415:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1076 00000100:00080000:2.0:1789761096.587417:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761096.587423:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761096.587424:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496001024:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789761096.587427:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761096.587436:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496001280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761096.587443:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761096.587445:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761096.587446:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761096.587449:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1076 00000100:00080000:2.0:1789761096.587452:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761096.587455:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761096.587458:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761096.587484:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496001408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761096.587517:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761096.587559:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761096/real 1789761096] req@ffff8dd205264380 x1876699496001024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761096.587575:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496001024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761096.587584:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761096.587588:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761096.587591:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496001024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761096.587598:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496001152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761096.587614:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761096.587623:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761096/real 1789761096] req@ffff8dd205264000 x1876699496001152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761096.587632:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496001152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761096.587639:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761096.587641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761096.587644:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496001152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761096.587665:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496001280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761096.587678:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761096.587685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761096/real 1789761096] req@ffff8dd205265880 x1876699496001280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761096.587694:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496001280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761096.587700:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761096.587702:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761096.587705:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496001280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761096.587709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496001408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761096.587720:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761096.587727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761096/real 1789761096] req@ffff8dd205266a00 x1876699496001408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761096.587735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496001408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761096.587741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761096.587743:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761096.587745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496001408:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761097.163293:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761098.187351:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761099.211333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761100.235325:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761101.259300:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761101.707329:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761101.707337:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761101.707340:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761101.707349:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1081 00000100:00080000:0.0:1789761101.707358:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761101.707364:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761101.707400:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496001536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761101.707424:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761101.707428:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761101.707430:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761101.707434:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1081 00000100:00080000:0.0:1789761101.707438:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761101.707442:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761101.707451:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761101.707473:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496001664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761101.707473:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496001536:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761101.707484:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761101.707487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761101.707489:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761101.707493:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1081 00000100:00080000:0.0:1789761101.707496:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761101.707499:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761101.707504:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761101.707514:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496001792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761101.707523:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761101.707525:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761101.707527:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761101.707530:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1081 00000100:00080000:0.0:1789761101.707533:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761101.707534:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761101.707537:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761101.707541:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761101.707549:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496001920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761101.707582:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761101/real 1789761101] req@ffff8dd211835c00 x1876699496001536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761101.707602:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496001536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761101.707611:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761101.707613:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761101.707616:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496001536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761101.707636:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496001664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761101.707655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761101.707664:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761101/real 1789761101] req@ffff8dd211836a00 x1876699496001664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761101.707672:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496001664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761101.707677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761101.707679:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761101.707681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496001664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761101.707685:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496001792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761101.707696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761101.707701:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761101/real 1789761101] req@ffff8dd211834000 x1876699496001792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761101.707709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496001792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761101.707718:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761101.707720:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761101.707722:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496001792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761101.707725:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496001920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761101.707734:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761101.707740:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761101/real 1789761101] req@ffff8dd211834700 x1876699496001920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761101.707746:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496001920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761101.707751:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761101.707753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761101.707755:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496001920:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761102.283287:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761103.307316:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761104.331310:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761105.355323:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761106.379369:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761106.827262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761106.827270:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761106.827274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761106.827283:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1086 00000100:00080000:0.0:1789761106.827292:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761106.827300:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761106.827331:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496002048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761106.827354:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761106.827356:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761106.827358:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761106.827361:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1086 00000100:00080000:0.0:1789761106.827363:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761106.827365:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761106.827372:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761106.827387:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496002176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761106.827398:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761106.827401:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761106.827402:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761106.827406:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1086 00000100:00080000:0.0:1789761106.827408:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761106.827410:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761106.827414:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761106.827422:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496002304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761106.827431:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761106.827435:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789761106.827435:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496002048:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761106.827441:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761106.827456:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1086 00000100:00080000:0.0:1789761106.827459:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761106.827461:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761106.827464:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761106.827473:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496002432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761106.827527:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761106.827573:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761106/real 1789761106] req@ffff8dd211835f80 x1876699496002048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761106.827589:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496002048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761106.827598:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761106.827602:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761106.827605:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496002048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761106.827612:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496002176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761106.827627:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761106.827635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761106/real 1789761106] req@ffff8dd211834380 x1876699496002176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761106.827644:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496002176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761106.827649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761106.827651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761106.827653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496002176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761106.827679:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496002304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761106.827691:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761106.827697:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761106/real 1789761106] req@ffff8dd211836680 x1876699496002304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761106.827705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496002304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761106.827710:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761106.827712:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761106.827714:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496002304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761106.827718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496002432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761106.827728:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761106.827734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761106/real 1789761106] req@ffff8dd211834a80 x1876699496002432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761106.827742:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496002432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761106.827747:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761106.827749:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761106.827751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496002432:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761107.403293:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761108.427344:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761109.451338:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761110.475307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761111.499313:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761111.947252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761111.947264:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761111.947268:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761111.947279:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1091 00000100:00080000:0.0:1789761111.947291:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761111.947300:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761111.947342:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496002560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761111.947368:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761111.947371:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761111.947373:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761111.947386:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1091 00000100:00080000:0.0:1789761111.947390:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761111.947393:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761111.947402:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761111.947413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496002688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761111.947423:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761111.947426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761111.947428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761111.947432:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496002560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761111.947434:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1091 00000100:00080000:0.0:1789761111.947440:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761111.947444:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761111.947448:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761111.947459:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496002816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761111.947475:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761111.947478:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761111.947480:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761111.947482:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1091 00000100:00080000:0.0:1789761111.947489:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761111.947491:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761111.947495:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761111.947501:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761111.947509:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496002944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761111.947544:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761111/real 1789761111] req@ffff8dd211837480 x1876699496002560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761111.947559:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496002560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761111.947568:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761111.947571:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761111.947575:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496002560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761111.947595:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496002688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761111.947620:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761111.947635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761111/real 1789761111] req@ffff8dd211835180 x1876699496002688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761111.947651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496002688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761111.947666:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761111.947669:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761111.947671:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496002688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761111.947676:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496002816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761111.947693:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761111.947701:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761111/real 1789761111] req@ffff8dd211836d80 x1876699496002816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761111.947710:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496002816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761111.947716:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761111.947718:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761111.947721:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496002816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761111.947724:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496002944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761111.947736:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761111.947743:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761111/real 1789761111] req@ffff8dd211834e00 x1876699496002944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761111.947751:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496002944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761111.947756:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761111.947758:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761111.947761:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496002944:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761112.523289:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761113.547269:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761114.571374:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761115.595343:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761116.619346:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761117.067282:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761117.067293:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761117.067297:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761117.067309:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1096 00000100:00080000:0.0:1789761117.067321:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761117.067328:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761117.067375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496003072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761117.067401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761117.067404:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761117.067406:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761117.067409:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1096 00000100:00080000:0.0:1789761117.067411:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761117.067414:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761117.067424:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761117.067438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496003200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761117.067446:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496003072:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761117.067456:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761117.067461:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761117.067463:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761117.067469:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1096 00000100:00080000:0.0:1789761117.067473:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761117.067478:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761117.067486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761117.067498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496003328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761117.067506:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761117.067509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761117.067511:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761117.067513:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1096 00000100:00080000:0.0:1789761117.067517:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761117.067521:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761117.067524:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761117.067529:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761117.067538:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496003456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761117.067577:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761117/real 1789761117] req@ffff8dd211834e00 x1876699496003072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761117.067592:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496003072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761117.067601:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761117.067604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761117.067608:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496003072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761117.067633:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496003200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761117.067650:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761117.067659:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761117/real 1789761117] req@ffff8dd211836d80 x1876699496003200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761117.067667:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496003200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761117.067674:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761117.067676:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761117.067678:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496003200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761117.067682:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496003328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761117.067694:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761117.067701:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761117/real 1789761117] req@ffff8dd211835180 x1876699496003328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761117.067709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496003328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761117.067714:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761117.067716:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761117.067718:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496003328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761117.067721:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496003456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761117.067731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761117.067737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761117/real 1789761117] req@ffff8dd211837480 x1876699496003456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761117.067745:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496003456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761117.067750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761117.067752:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761117.067754:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496003456:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761117.643356:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761118.667310:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761119.691354:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761120.715358:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761121.739360:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761122.187264:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761122.187272:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761122.187276:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761122.187285:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1101 00000100:00080000:0.0:1789761122.187296:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761122.187303:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761122.187341:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496003584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761122.187370:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761122.187372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761122.187374:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761122.187378:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1101 00000100:00080000:0.0:1789761122.187382:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761122.187386:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761122.187396:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761122.187412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496003712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761122.187425:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761122.187428:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761122.187433:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761122.187434:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496003584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761122.187440:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1101 00000100:00080000:0.0:1789761122.187446:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761122.187451:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761122.187458:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761122.187470:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496003840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761122.187482:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761122.187496:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761122.187497:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761122.187501:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1101 00000100:00080000:0.0:1789761122.187514:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761122.187520:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761122.187524:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789761122.187528:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761122.187545:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496003968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761122.187582:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761122/real 1789761122] req@ffff8dd211834a80 x1876699496003584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761122.187601:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496003584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761122.187610:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761122.187613:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761122.187616:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496003584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761122.187653:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496003712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761122.187675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761122.187685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761122/real 1789761122] req@ffff8dd211836680 x1876699496003712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761122.187695:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496003712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761122.187702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761122.187704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761122.187707:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496003712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761122.187711:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496003840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761122.187724:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761122.187730:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761122/real 1789761122] req@ffff8dd211834380 x1876699496003840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761122.187739:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496003840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761122.187745:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761122.187747:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761122.187750:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496003840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761122.187754:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496003968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761122.187803:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761122.187811:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761122/real 1789761122] req@ffff8dd211835f80 x1876699496003968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761122.187819:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496003968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761122.187824:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761122.187827:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761122.187829:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496003968:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761122.763316:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761123.787350:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761124.811323:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761125.835386:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761126.859354:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761127.307248:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761127.307259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761127.307264:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761127.307276:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1106 00000100:00080000:0.0:1789761127.307295:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761127.307303:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761127.307352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496004096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761127.307385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761127.307388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761127.307390:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761127.307394:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1106 00000100:00080000:0.0:1789761127.307399:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761127.307404:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761127.307414:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761127.307429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496004224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761127.307435:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496004096:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761127.307440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761127.307449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761127.307450:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761127.307455:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1106 00000100:00080000:0.0:1789761127.307458:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761127.307461:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761127.307464:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761127.307478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496004352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761127.307487:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761127.307490:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761127.307492:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761127.307496:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1106 00000100:00080000:0.0:1789761127.307511:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761127.307514:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761127.307518:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761127.307524:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761127.307528:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496004480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761127.307580:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761127/real 1789761127] req@ffff8dd211834700 x1876699496004096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761127.307596:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496004096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761127.307605:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761127.307609:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761127.307612:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496004096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761127.307632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496004224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761127.307652:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761127.307661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761127/real 1789761127] req@ffff8dd211834000 x1876699496004224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761127.307670:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496004224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761127.307676:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761127.307678:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761127.307680:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496004224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761127.307684:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496004352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761127.307696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761127.307702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761127/real 1789761127] req@ffff8dd211836a00 x1876699496004352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761127.307709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496004352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761127.307715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761127.307717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761127.307719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496004352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761127.307738:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496004480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761127.307753:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761127.307761:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761127/real 1789761127] req@ffff8dd211835c00 x1876699496004480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761127.307769:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496004480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761127.307779:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761127.307781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761127.307783:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496004480:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761127.883354:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761128.907292:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761129.931382:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761130.955294:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761131.979329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761132.427284:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761132.427295:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761132.427300:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761132.427310:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1112 00000100:00080000:2.0:1789761132.427322:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761132.427338:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761132.427370:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496004608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761132.427399:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761132.427403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761132.427405:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761132.427410:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1112 00000100:00080000:2.0:1789761132.427422:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761132.427428:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761132.427439:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761132.427448:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496004608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761132.427457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496004736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761132.427465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761132.427468:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761132.427470:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761132.427472:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1112 00000100:00080000:2.0:1789761132.427475:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761132.427478:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761132.427482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761132.427491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496004864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761132.427498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761132.427501:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761132.427503:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761132.427505:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1112 00000100:00080000:2.0:1789761132.427508:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761132.427511:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761132.427524:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761132.427528:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761132.427542:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496004992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761132.427574:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761132/real 1789761132] req@ffff8dd205264700 x1876699496004608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761132.427590:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496004608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761132.427598:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761132.427601:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761132.427604:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496004608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761132.427623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496004736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761132.427641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761132.427649:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761132/real 1789761132] req@ffff8dd205266d80 x1876699496004736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761132.427657:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496004736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761132.427663:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761132.427665:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761132.427667:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496004736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761132.427672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496004864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761132.427700:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761132.427711:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761132/real 1789761132] req@ffff8dd205265f80 x1876699496004864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761132.427719:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496004864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761132.427725:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761132.427727:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761132.427729:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496004864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761132.427733:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496004992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761132.427747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761132.427753:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761132/real 1789761132] req@ffff8dd205266300 x1876699496004992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761132.427760:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496004992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761132.427766:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761132.427767:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761132.427769:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496004992:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761133.003335:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761134.027286:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761135.051302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761136.075352:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761137.099323:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761137.547279:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761137.547287:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761137.547291:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761137.547300:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1117 00000100:00080000:2.0:1789761137.547309:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761137.547316:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761137.547356:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496005120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761137.547385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761137.547388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761137.547389:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761137.547393:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1117 00000100:00080000:2.0:1789761137.547396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761137.547399:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761137.547407:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761137.547422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496005248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761137.547432:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496005120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761137.547450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761137.547454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761137.547456:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761137.547462:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1117 00000100:00080000:2.0:1789761137.547467:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761137.547471:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761137.547477:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761137.547493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496005376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761137.547496:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761137.547510:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761137.547514:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761137.547517:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761137.547522:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1117 00000100:00080000:2.0:1789761137.547527:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761137.547532:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761137.547538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761137/real 1789761137] req@ffff8dd205265180 x1876699496005120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761137.547539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761137.547559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496005504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761137.547560:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496005120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761137.547593:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761137.547597:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761137.547601:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496005120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761137.547621:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496005248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761137.547637:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761137.547665:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761137/real 1789761137] req@ffff8dd205266300 x1876699496005248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761137.547675:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496005248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761137.547681:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761137.547683:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761137.547686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496005248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761137.547692:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496005376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761137.547714:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761137.547727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761137/real 1789761137] req@ffff8dd205265f80 x1876699496005376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761137.547740:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496005376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761137.547752:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761137.547755:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761137.547757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496005376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761137.547761:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496005504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761137.547776:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761137.547785:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761137/real 1789761137] req@ffff8dd205266d80 x1876699496005504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761137.547795:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496005504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761137.547800:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761137.547802:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761137.547804:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496005504:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761138.123306:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761139.147325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761140.171324:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761141.195355:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761142.219321:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761142.667291:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761142.667302:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761142.667307:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761142.667319:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1122 00000100:00080000:2.0:1789761142.667331:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761142.667339:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761142.667392:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496005632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761142.667427:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761142.667431:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761142.667433:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761142.667437:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1122 00000100:00080000:2.0:1789761142.667442:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761142.667447:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761142.667460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761142.667478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496005760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761142.667485:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496005632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761142.667507:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761142.667512:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761142.667515:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761142.667523:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1122 00000100:00080000:2.0:1789761142.667528:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761142.667533:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761142.667539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761142.667554:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496005888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761142.667575:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761142.667578:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761142.667584:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761142.667586:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761142.667592:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1122 00000100:00080000:2.0:1789761142.667599:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761142.667604:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761142.667624:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761142.667641:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761142/real 1789761142] req@ffff8dd205264700 x1876699496005632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761142.667643:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496006016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761142.667709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496005632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761142.667719:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761142.667729:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761142.667733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496005632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761142.667756:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496005760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761142.667777:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761142.667787:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761142/real 1789761142] req@ffff8dd205266a00 x1876699496005760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761142.667796:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496005760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761142.667803:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761142.667805:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761142.667808:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496005760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761142.667812:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496005888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761142.667825:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761142.667831:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761142/real 1789761142] req@ffff8dd205265880 x1876699496005888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761142.667838:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496005888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761142.667844:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761142.667846:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761142.667848:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496005888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761142.667851:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496006016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761142.667862:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761142.667868:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761142/real 1789761142] req@ffff8dd205264000 x1876699496006016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761142.667876:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496006016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761142.667881:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761142.667883:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761142.667885:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496006016:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761143.243337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761144.267322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761145.291405:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761146.315349:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761147.339324:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761147.787300:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761147.787315:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761147.787320:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761147.787333:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1127 00000100:00080000:2.0:1789761147.787348:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761147.787357:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761147.787407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496006144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761147.787442:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761147.787447:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761147.787450:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761147.787456:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1127 00000100:00080000:2.0:1789761147.787462:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761147.787467:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761147.787479:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761147.787489:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496006144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761147.787503:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496006272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761147.787520:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761147.787524:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761147.787526:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761147.787532:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1127 00000100:00080000:2.0:1789761147.787537:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761147.787542:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761147.787551:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761147.787582:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496006400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761147.787598:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761147.787602:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761147.787605:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761147.787610:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1127 00000100:00100000:1.0:1789761147.787617:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496006272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1789761147.787620:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761147.787632:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761147.787641:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761147.787661:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496006528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761147.787670:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496006400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761147.787712:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496006528:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761148.363300:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761149.387370:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761150.411357:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761150.859479:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789761150.859567:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761150.859593:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761150.859646:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761150.859661:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761150.859693:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789761150.859699:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761147/real 1789761150] req@ffff8dd205267b80 x1876699496006272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761150.859733:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496006272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761150.859746:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761150.859751:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761150.859756:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496006272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761150.859764:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761147/real 1789761150] req@ffff8dd205264e00 x1876699496006400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761150.859778:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496006400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761150.859787:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761150.859791:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761150.859795:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496006400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761150.859816:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761147/real 1789761150] req@ffff8dd205264380 x1876699496006144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761150.859834:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496006144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761150.859846:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761150.859849:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761150.859852:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496006144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761150.859881:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761147/real 1789761150] req@ffff8dd205267800 x1876699496006528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761150.859893:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496006528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761150.859900:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761150.859903:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761150.859907:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496006528:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761151.435349:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761152.459291:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761152.907322:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761152.907336:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761152.907341:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761152.907354:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1132 00000100:00080000:0.0:1789761152.907369:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761152.907378:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761152.907427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496006656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761152.907464:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761152.907469:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761152.907471:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761152.907476:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1132 00000100:00080000:0.0:1789761152.907481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761152.907486:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761152.907498:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761152.907514:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496006784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761152.907526:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761152.907546:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761152.907549:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761152.907555:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1132 00000100:00080000:0.0:1789761152.907559:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761152.907565:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761152.907572:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761152.907593:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496006912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761152.907595:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496006656:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761152.907623:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761152.907627:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761152.907630:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761152.907636:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1132 00000100:00080000:0.0:1789761152.907641:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761152.907646:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761152.907654:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761152.907675:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496007040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761152.907697:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761152.907749:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761152/real 1789761152] req@ffff8dd211837b80 x1876699496006656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761152.907767:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496006656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761152.907778:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761152.907782:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761152.907785:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496006656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761152.907795:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496006784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761152.907820:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761152.907849:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761152/real 1789761152] req@ffff8dd211837100 x1876699496006784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761152.907862:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496006784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761152.907870:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761152.907874:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761152.907877:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496006784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761152.907905:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496006912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761152.907928:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761152.907939:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761152/real 1789761152] req@ffff8dd211836300 x1876699496006912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761152.907950:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496006912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761152.907957:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761152.907960:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761152.907963:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496006912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761152.907969:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496007040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761152.907989:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761152.908000:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761152/real 1789761152] req@ffff8dd211837800 x1876699496007040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761152.908012:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496007040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761152.908020:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761152.908022:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761152.908026:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496007040:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761153.483328:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761154.507343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761155.531365:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761156.555378:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761157.579326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761158.027234:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761158.027246:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761158.027251:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761158.027261:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1137 00000100:00080000:0.0:1789761158.027273:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761158.027281:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761158.027320:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496007168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761158.027353:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761158.027361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761158.027363:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761158.027367:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1137 00000100:00080000:0.0:1789761158.027370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761158.027376:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761158.027376:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496007168:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789761158.027390:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761158.027404:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496007296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761158.027412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761158.027415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761158.027416:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761158.027419:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1137 00000100:00080000:0.0:1789761158.027421:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761158.027424:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761158.027428:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761158.027436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496007424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761158.027440:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761158.027452:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761158.027457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761158.027459:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761158.027464:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1137 00000100:00080000:0.0:1789761158.027468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761158.027473:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761158.027478:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761158/real 1789761158] req@ffff8dd211837800 x1876699496007168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789761158.027479:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761158.027511:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496007552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761158.027514:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496007168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761158.027536:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761158.027539:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761158.027543:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496007168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761158.027566:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496007296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761158.027584:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761158.027593:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761158/real 1789761158] req@ffff8dd211836300 x1876699496007296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761158.027602:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496007296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761158.027613:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761158.027616:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761158.027620:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496007296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761158.027626:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496007424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761158.027647:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761158.027658:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761158/real 1789761158] req@ffff8dd211837100 x1876699496007424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761158.027672:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496007424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761158.027681:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761158.027684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761158.027688:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496007424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761158.027694:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496007552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761158.027715:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761158.027727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761158/real 1789761158] req@ffff8dd211837b80 x1876699496007552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761158.027741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496007552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761158.027750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761158.027753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761158.027757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496007552:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761158.603311:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761159.627301:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761160.651325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761161.675327:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761162.699330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761163.147340:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761163.147351:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761163.147356:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761163.147366:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1142 00000100:00080000:2.0:1789761163.147378:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761163.147385:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761163.147425:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496007680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761163.147469:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761163.147473:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761163.147475:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761163.147480:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1142 00000100:00080000:2.0:1789761163.147484:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761163.147488:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761163.147505:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761163.147522:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496007680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761163.147523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496007808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761163.147548:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761163.147552:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761163.147554:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761163.147559:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1142 00000100:00080000:2.0:1789761163.147573:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761163.147577:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761163.147581:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789761163.147586:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761163.147595:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496007936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761163.147605:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761163.147607:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761163.147608:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761163.147611:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1142 00000100:00080000:2.0:1789761163.147614:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761163.147617:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761163.147621:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761163.147631:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496008064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761163.147633:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761163/real 1789761163] req@ffff8dd205267100 x1876699496007680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761163.147661:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496007680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761163.147679:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761163.147683:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761163.147688:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496007680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761163.147711:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496007808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761163.147735:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761163.147747:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761163/real 1789761163] req@ffff8dd205266680 x1876699496007808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761163.147760:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496007808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761163.147769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761163.147773:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761163.147776:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496007808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761163.147783:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496007936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761163.147804:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761163.147817:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761163/real 1789761163] req@ffff8dd205265500 x1876699496007936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761163.147830:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496007936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761163.147840:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761163.147843:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761163.147847:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496007936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761163.147853:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496008064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761163.147874:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761163.147885:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761163/real 1789761163] req@ffff8dd205264a80 x1876699496008064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761163.147897:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496008064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761163.147906:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761163.147909:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761163.147913:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496008064:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761163.723346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761164.747291:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761165.771295:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761166.795318:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761167.819294:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761168.267245:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761168.267274:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761168.267282:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761168.267292:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1147 00000100:00080000:2.0:1789761168.267303:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761168.267310:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761168.267339:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496008192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761168.267358:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761168.267361:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761168.267363:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761168.267366:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1147 00000100:00080000:2.0:1789761168.267368:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761168.267370:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761168.267378:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761168.267389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496008320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761168.267407:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761168.267411:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761168.267414:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761168.267414:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496008192:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761168.267434:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1147 00000100:00080000:2.0:1789761168.267439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761168.267442:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761168.267447:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761168.267458:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496008448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761168.267476:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761168.267479:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761168.267480:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761168.267483:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1147 00000100:00080000:2.0:1789761168.267486:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761168.267489:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761168.267491:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761168.267501:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761168.267502:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496008576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761168.267545:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761168/real 1789761168] req@ffff8dd205267480 x1876699496008192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761168.267564:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496008192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761168.267575:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761168.267579:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761168.267583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496008192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761168.267605:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496008320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761168.267628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761168.267642:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761168/real 1789761168] req@ffff8dd205264a80 x1876699496008320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761168.267655:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496008320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761168.267663:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761168.267667:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761168.267670:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496008320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761168.267676:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496008448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761168.267695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761168.267706:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761168/real 1789761168] req@ffff8dd205265500 x1876699496008448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761168.267717:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496008448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761168.267726:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761168.267729:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761168.267733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496008448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761168.267739:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496008576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761168.267756:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761168.267766:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761168/real 1789761168] req@ffff8dd205266680 x1876699496008576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761168.267777:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496008576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761168.267785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761168.267788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761168.267791:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496008576:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761168.843322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761169.867295:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761170.891353:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761171.915288:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761172.939312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761173.387270:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761173.387281:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761173.387285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761173.387296:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1152 00000100:00080000:2.0:1789761173.387308:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761173.387315:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761173.387365:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496008704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761173.387410:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761173.387415:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761173.387417:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761173.387423:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1152 00000100:00080000:2.0:1789761173.387427:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761173.387433:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761173.387445:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761173.387464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496008832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761173.387478:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761173.387481:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496008704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761173.387485:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761173.387492:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761173.387500:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1152 00000100:00080000:2.0:1789761173.387505:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761173.387511:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761173.387518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761173.387534:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496008960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761173.387556:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789761173.387557:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761173.387566:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761173.387570:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761173.387576:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1152 00000100:00080000:2.0:1789761173.387583:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761173.387587:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761173.387594:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761173.387607:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496009088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761173.387620:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761173/real 1789761173] req@ffff8dd205267100 x1876699496008704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761173.387647:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496008704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761173.387660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761173.387665:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761173.387669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496008704:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761173.387697:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496008832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761173.387726:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761173.387741:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761173/real 1789761173] req@ffff8dd205267800 x1876699496008832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761173.387756:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496008832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761173.387776:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761173.387778:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761173.387781:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496008832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761173.387786:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496008960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761173.387804:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761173.387812:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761173/real 1789761173] req@ffff8dd205264380 x1876699496008960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761173.387820:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496008960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761173.387826:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761173.387839:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761173.387841:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496008960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761173.387845:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496009088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761173.387855:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761173.387861:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761173/real 1789761173] req@ffff8dd205264e00 x1876699496009088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761173.387868:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496009088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761173.387878:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761173.387880:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761173.387882:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496009088:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761173.963357:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761174.987319:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761176.011408:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761177.035386:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761178.059359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761178.507299:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761178.507307:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761178.507311:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761178.507320:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1158 00000100:00080000:0.0:1789761178.507347:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761178.507356:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761178.507394:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496009216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761178.507424:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761178.507426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761178.507428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761178.507431:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1158 00000100:00080000:0.0:1789761178.507434:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761178.507436:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761178.507449:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761178.507461:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496009344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761178.507468:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761178.507471:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761178.507472:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761178.507475:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1158 00000100:00080000:0.0:1789761178.507478:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761178.507480:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761178.507485:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496009216:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789761178.507486:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761178.507498:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496009472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761178.507506:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761178.507509:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761178.507510:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761178.507513:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1158 00000100:00080000:0.0:1789761178.507516:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761178.507518:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761178.507522:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761178.507529:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496009600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761178.507548:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761178.507586:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761178/real 1789761178] req@ffff8dd211835c00 x1876699496009216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761178.507598:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496009216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761178.507607:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761178.507610:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761178.507613:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496009216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761178.507618:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496009344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761178.507633:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761178.507641:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761178/real 1789761178] req@ffff8dd211836a00 x1876699496009344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761178.507649:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496009344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761178.507654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761178.507656:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761178.507658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496009344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761178.507676:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496009472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761178.507687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761178.507693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761178/real 1789761178] req@ffff8dd211834000 x1876699496009472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761178.507700:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496009472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761178.507706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761178.507707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761178.507709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496009472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761178.507713:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496009600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761178.507722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761178.507728:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761178/real 1789761178] req@ffff8dd211834700 x1876699496009600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761178.507736:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496009600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761178.507741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761178.507742:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761178.507744:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496009600:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761179.083385:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761180.107286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761181.131337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761182.155338:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761183.179315:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761183.627295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761183.627314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761183.627317:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761183.627327:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1163 00000100:00080000:1.0:1789761183.627336:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761183.627342:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761183.627374:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496009728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761183.627396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761183.627400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761183.627401:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761183.627406:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1163 00000100:00080000:1.0:1789761183.627409:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761183.627414:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761183.627423:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761183.627436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496009856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761183.627445:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496009728:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761183.627446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761183.627450:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761183.627451:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761183.627455:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1163 00000100:00080000:1.0:1789761183.627458:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761183.627462:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761183.627466:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761183.627484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496009984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761183.627494:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761183.627497:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761183.627498:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761183.627501:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1163 00000100:00080000:1.0:1789761183.627504:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761183.627504:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761183.627508:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761183.627512:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761183.627521:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496010112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761183.627544:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761183/real 1789761183] req@ffff8dd20504ce00 x1876699496009728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761183.627563:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496009728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761183.627572:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761183.627575:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761183.627578:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496009728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761183.627596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496009856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761183.627615:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761183.627623:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761183/real 1789761183] req@ffff8dd20504f800 x1876699496009856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761183.627631:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496009856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761183.627637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761183.627639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761183.627641:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496009856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761183.627645:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496009984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761183.627657:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761183.627663:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761183/real 1789761183] req@ffff8dd20504fb80 x1876699496009984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761183.627671:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496009984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761183.627676:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761183.627678:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761183.627680:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496009984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761183.627683:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496010112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761183.627693:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761183.627699:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761183/real 1789761183] req@ffff8dd20504df80 x1876699496010112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761183.627706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496010112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761183.627711:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761183.627713:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761183.627715:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496010112:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761184.203341:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761185.227314:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761186.251346:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761187.275313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761188.299329:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761188.747258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761188.747283:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761188.747288:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761188.747299:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1168 00000100:00080000:1.0:1789761188.747310:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761188.747326:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761188.747360:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496010240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761188.747388:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761188.747391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761188.747392:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761188.747396:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1168 00000100:00080000:1.0:1789761188.747398:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761188.747401:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761188.747413:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761188.747426:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496010368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761188.747434:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761188.747435:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496010240:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761188.747441:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761188.747444:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761188.747447:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1168 00000100:00080000:1.0:1789761188.747450:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761188.747452:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761188.747456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761188.747465:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496010496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761188.747473:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761188.747476:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761188.747477:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761188.747479:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1168 00000100:00080000:1.0:1789761188.747488:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761188.747490:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761188.747494:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761188.747498:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761188.747506:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496010624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761188.747536:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761188/real 1789761188] req@ffff8dd20504d500 x1876699496010240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761188.747550:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496010240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761188.747557:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761188.747560:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761188.747563:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496010240:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761188.747582:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496010368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761188.747596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761188.747604:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761188/real 1789761188] req@ffff8dd20504c700 x1876699496010368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761188.747623:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496010368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761188.747633:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761188.747636:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761188.747639:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496010368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761188.747645:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496010496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761188.747667:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761188.747680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761188/real 1789761188] req@ffff8dd20504e680 x1876699496010496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761188.747693:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496010496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761188.747702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761188.747705:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761188.747709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496010496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761188.747715:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496010624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761188.747739:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761188.747753:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761188/real 1789761188] req@ffff8dd20504d180 x1876699496010624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761188.747765:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496010624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761188.747780:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761188.747784:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761188.747787:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496010624:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761189.323295:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761190.347395:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761191.371282:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761192.395371:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761193.419317:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761193.867356:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761193.867367:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761193.867372:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761193.867383:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1173 00000100:00080000:0.0:1789761193.867395:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761193.867402:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761193.867443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496010752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761193.867473:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761193.867477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761193.867478:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761193.867482:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1173 00000100:00080000:0.0:1789761193.867484:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761193.867486:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761193.867495:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761193.867512:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496010880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761193.867520:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761193.867523:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496010752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761193.867525:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761193.867529:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761193.867534:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1173 00000100:00080000:0.0:1789761193.867536:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761193.867539:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761193.867543:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761193.867553:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496011008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761193.867561:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761193.867564:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761193.867566:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761193.867568:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1173 00000100:00080000:0.0:1789761193.867571:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761193.867573:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761193.867576:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761193.867583:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496011136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761193.867597:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761193.867640:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761193/real 1789761193] req@ffff8dd211835f80 x1876699496010752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761193.867656:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496010752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761193.867664:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761193.867668:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761193.867673:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496010752:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761193.867701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496010880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761193.867728:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761193.867742:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761193/real 1789761193] req@ffff8dd211834380 x1876699496010880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761193.867756:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496010880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761193.867766:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761193.867770:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761193.867774:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496010880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761193.867780:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496011008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761193.867819:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761193.867832:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761193/real 1789761193] req@ffff8dd211836680 x1876699496011008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761193.867841:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496011008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761193.867852:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761193.867855:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761193.867857:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496011008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761193.867861:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496011136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761193.867877:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761193.867883:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761193/real 1789761193] req@ffff8dd211834a80 x1876699496011136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761193.867891:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496011136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761193.867896:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761193.867898:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761193.867900:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496011136:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761194.443351:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761195.467331:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761196.491346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761197.515318:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761198.539327:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761198.987255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761198.987263:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761198.987266:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761198.987274:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1178 00000100:00080000:0.0:1789761198.987283:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761198.987288:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761198.987318:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496011264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761198.987347:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761198.987350:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761198.987351:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761198.987354:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1178 00000100:00080000:0.0:1789761198.987356:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761198.987359:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761198.987366:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761198.987374:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496011392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761198.987382:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761198.987384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761198.987386:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761198.987389:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1178 00000100:00080000:0.0:1789761198.987391:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761198.987396:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761198.987396:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496011264:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789761198.987404:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761198.987413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496011520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761198.987421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761198.987424:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761198.987425:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761198.987428:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1178 00000100:00080000:0.0:1789761198.987430:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761198.987432:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761198.987435:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761198.987441:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496011648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761198.987468:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761198.987506:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761198/real 1789761198] req@ffff8dd211837480 x1876699496011264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761198.987519:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496011264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761198.987527:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761198.987530:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761198.987533:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496011264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761198.987541:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496011392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761198.987560:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761198.987569:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761198/real 1789761198] req@ffff8dd211835180 x1876699496011392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761198.987578:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496011392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761198.987584:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761198.987586:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761198.987588:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496011392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761198.987607:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496011520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761198.987619:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761198.987625:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761198/real 1789761198] req@ffff8dd211836d80 x1876699496011520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761198.987633:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496011520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761198.987638:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761198.987640:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761198.987642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496011520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761198.987646:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496011648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761198.987655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761198.987670:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761198/real 1789761198] req@ffff8dd211834e00 x1876699496011648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761198.987678:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496011648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761198.987688:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761198.987690:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761198.987694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496011648:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761199.563338:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761200.587327:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761201.611331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761202.635335:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761203.659373:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761204.107317:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761204.107329:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761204.107333:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761204.107345:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1183 00000100:00080000:2.0:1789761204.107357:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761204.107364:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761204.107408:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496011776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761204.107434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761204.107438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761204.107439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761204.107442:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1183 00000100:00080000:2.0:1789761204.107445:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761204.107449:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761204.107459:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761204.107471:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496011904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761204.107474:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496011776:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761204.107488:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761204.107491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761204.107492:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761204.107496:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1183 00000100:00080000:2.0:1789761204.107499:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761204.107501:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761204.107505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761204.107513:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496012032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761204.107520:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761204.107523:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761204.107524:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761204.107527:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1183 00000100:00080000:2.0:1789761204.107529:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761204.107532:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761204.107534:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761204.107541:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496012160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761204.107553:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761204.107599:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761204/real 1789761204] req@ffff8dd205267b80 x1876699496011776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761204.107615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496011776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761204.107623:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761204.107626:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761204.107630:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496011776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761204.107651:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496011904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761204.107667:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761204.107675:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761204/real 1789761204] req@ffff8dd205264000 x1876699496011904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761204.107683:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496011904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761204.107689:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761204.107691:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761204.107693:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496011904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761204.107697:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496012032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761204.107709:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761204.107715:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761204/real 1789761204] req@ffff8dd205265880 x1876699496012032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761204.107723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496012032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761204.107728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761204.107730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761204.107732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496012032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761204.107735:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496012160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761204.107747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761204.107753:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761204/real 1789761204] req@ffff8dd205266a00 x1876699496012160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761204.107760:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496012160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761204.107765:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761204.107767:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761204.107769:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496012160:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761204.683333:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761205.707353:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761206.731289:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761207.755322:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761208.779295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761209.227238:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761209.227250:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761209.227254:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761209.227266:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1188 00000100:00080000:2.0:1789761209.227279:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761209.227287:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761209.227330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496012288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761209.227356:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761209.227359:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761209.227361:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761209.227366:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1188 00000100:00080000:2.0:1789761209.227368:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761209.227371:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761209.227382:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761209.227393:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496012416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761209.227402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761209.227404:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761209.227406:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761209.227408:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1188 00000100:00080000:2.0:1789761209.227411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761209.227414:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761209.227419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761209.227427:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496012288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761209.227430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496012544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761209.227455:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761209.227457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761209.227458:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761209.227461:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1188 00000100:00080000:2.0:1789761209.227464:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761209.227467:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761209.227469:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761209.227479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496012672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761209.227522:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761209.227563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761209/real 1789761209] req@ffff8dd205264700 x1876699496012288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761209.227577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496012288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761209.227586:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761209.227589:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761209.227593:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496012288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761209.227614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496012416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761209.227631:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761209.227639:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761209/real 1789761209] req@ffff8dd205266d80 x1876699496012416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761209.227648:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496012416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761209.227654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761209.227656:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761209.227658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496012416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761209.227664:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496012544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761209.227685:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761209.227698:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761209/real 1789761209] req@ffff8dd205265f80 x1876699496012544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761209.227712:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496012544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761209.227728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761209.227731:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761209.227734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496012544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761209.227739:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496012672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761209.227762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761209.227775:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761209/real 1789761209] req@ffff8dd205266300 x1876699496012672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761209.227789:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496012672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761209.227798:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761209.227801:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761209.227803:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496012672:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761209.803325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761210.827268:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761211.851329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761212.875339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761213.899326:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761214.347277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761214.347286:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761214.347291:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761214.347301:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1193 00000100:00080000:2.0:1789761214.347314:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761214.347322:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761214.347364:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496012800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761214.347395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761214.347398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761214.347399:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761214.347403:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1193 00000100:00080000:2.0:1789761214.347405:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761214.347407:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761214.347417:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761214.347427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496012928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761214.347435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761214.347438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761214.347439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761214.347442:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1193 00000100:00080000:2.0:1789761214.347444:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761214.347449:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761214.347453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761214.347461:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496013056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761214.347472:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761214.347476:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761214.347477:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761214.347481:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1193 00000100:00080000:2.0:1789761214.347485:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761214.347492:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496012800:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761214.347498:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761214.347506:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761214.347514:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496013184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761214.347619:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496012928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761214.347651:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496013056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761214.347675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496013184:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761214.923334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761215.947362:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761216.971366:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761217.419439:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789761217.419495:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761217.419517:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761217.419562:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761217.419571:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761217.419582:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789761217.419659:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761214/real 1789761217] req@ffff8dd205265f80 x1876699496013056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761217.419695:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496013056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761217.419709:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761217.419715:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761217.419720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496013056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761217.419730:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761214/real 1789761217] req@ffff8dd205266d80 x1876699496013184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761217.419742:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496013184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761217.419750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761217.419753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761217.419755:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496013184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761217.419763:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761214/real 1789761217] req@ffff8dd205265180 x1876699496012800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761217.419776:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496012800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761217.419782:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761217.419785:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761217.419789:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496012800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761217.419795:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761214/real 1789761217] req@ffff8dd205266300 x1876699496012928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761217.419807:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496012928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761217.419815:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761217.419819:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761217.419822:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496012928:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761217.995335:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761219.019341:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761219.467284:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761219.467293:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761219.467297:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761219.467305:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1199 00000100:00080000:1.0:1789761219.467315:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761219.467320:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761219.467357:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496013312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761219.467390:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761219.467394:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761219.467396:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761219.467399:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1199 00000100:00080000:1.0:1789761219.467402:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761219.467413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761219.467423:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761219.467432:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496013312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761219.467443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496013440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761219.467456:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761219.467460:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761219.467463:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761219.467468:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1199 00000100:00080000:1.0:1789761219.467472:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761219.467477:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761219.467483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789761219.467490:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761219.467505:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496013568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761219.467518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761219.467521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761219.467523:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761219.467530:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1199 00000100:00100000:3.0:1789761219.467530:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761219/real 1789761219] req@ffff8dd20504e300 x1876699496013312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761219.467538:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761219.467546:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761219.467552:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761219.467553:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496013312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761219.467568:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496013696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761219.467569:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761219.467580:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761219.467589:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496013312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761219.467619:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496013440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761219.467645:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761219.467671:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761219/real 1789761219] req@ffff8dd20504d180 x1876699496013440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761219.467684:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496013440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761219.467693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761219.467696:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761219.467700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496013440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761219.467706:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496013568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761219.467729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761219.467741:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761219/real 1789761219] req@ffff8dd20504e680 x1876699496013568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761219.467754:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496013568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761219.467762:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761219.467765:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761219.467768:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496013568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761219.467774:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496013696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761219.467796:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761219.467808:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761219/real 1789761219] req@ffff8dd20504c700 x1876699496013696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761219.467822:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496013696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761219.467831:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761219.467843:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761219.467850:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496013696:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761220.043337:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761221.067331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761222.091307:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761223.115315:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761224.139309:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761224.587254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761224.587268:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761224.587274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761224.587288:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1204 00000100:00080000:0.0:1789761224.587302:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761224.587312:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761224.587356:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496013824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761224.587381:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761224.587384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761224.587386:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761224.587391:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1204 00000100:00080000:0.0:1789761224.587396:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761224.587400:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761224.587412:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761224.587415:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496013824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761224.587431:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496013952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761224.587444:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761224.587447:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761224.587449:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761224.587452:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1204 00000100:00080000:0.0:1789761224.587456:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761224.587459:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761224.587464:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761224.587474:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496014080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761224.587483:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761224.587486:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761224.587487:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761224.587489:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1204 00000100:00080000:0.0:1789761224.587492:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761224.587497:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761224.587500:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761224.587508:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761224.587523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496014208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761224.587550:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761224/real 1789761224] req@ffff8dd211834e00 x1876699496013824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761224.587568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496013824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761224.587577:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761224.587580:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761224.587584:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496013824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761224.587609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496013952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761224.587632:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761224.587656:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761224/real 1789761224] req@ffff8dd211836d80 x1876699496013952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761224.587671:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496013952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761224.587684:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761224.587687:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761224.587689:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496013952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761224.587694:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496014080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761224.587712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761224.587720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761224/real 1789761224] req@ffff8dd211835180 x1876699496014080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761224.587728:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496014080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761224.587734:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761224.587737:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761224.587739:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496014080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761224.587742:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496014208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761224.587753:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761224.587759:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761224/real 1789761224] req@ffff8dd211837480 x1876699496014208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761224.587766:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496014208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761224.587771:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761224.587773:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761224.587776:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496014208:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761225.163325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761226.187327:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761227.211380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761228.235322:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761229.259338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761229.707297:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761229.707306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761229.707310:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761229.707319:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1209 00000100:00080000:1.0:1789761229.707329:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761229.707338:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761229.707376:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496014336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761229.707413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761229.707417:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761229.707419:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761229.707425:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1209 00000100:00080000:1.0:1789761229.707429:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761229.707434:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761229.707445:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761229.707453:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496014336:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761229.707464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496014464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761229.707477:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761229.707481:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761229.707483:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761229.707488:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1209 00000100:00080000:1.0:1789761229.707492:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761229.707497:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761229.707501:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761229.707512:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496014592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761229.707523:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789761229.707527:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761229.707529:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761229.707533:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761229.707538:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1209 00000100:00080000:1.0:1789761229.707544:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761229.707550:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761229.707556:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761229.707571:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496014720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761229.707574:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761229/real 1789761229] req@ffff8dd20504d500 x1876699496014336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761229.707610:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496014336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761229.707621:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761229.707625:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761229.707630:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496014336:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761229.707654:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496014464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761229.707680:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761229.707693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761229/real 1789761229] req@ffff8dd20504df80 x1876699496014464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761229.707704:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496014464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761229.707713:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761229.707716:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761229.707720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496014464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761229.707727:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496014592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761229.707746:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761229.707758:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761229/real 1789761229] req@ffff8dd20504fb80 x1876699496014592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761229.707769:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496014592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761229.707777:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761229.707781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761229.707783:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496014592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761229.707789:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496014720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761229.707806:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761229.707816:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761229/real 1789761229] req@ffff8dd20504f800 x1876699496014720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761229.707827:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496014720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761229.707835:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761229.707838:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761229.707841:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496014720:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761230.283287:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761231.307354:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761232.331329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761233.355316:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761234.379333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761234.827256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761234.827269:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761234.827273:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761234.827283:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1214 00000100:00080000:1.0:1789761234.827296:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761234.827303:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761234.827342:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496014848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761234.827364:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761234.827367:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761234.827369:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761234.827372:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1214 00000100:00080000:1.0:1789761234.827375:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761234.827377:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761234.827388:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761234.827397:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496014976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761234.827404:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496014848:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761234.827408:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761234.827413:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761234.827414:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761234.827417:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1214 00000100:00080000:1.0:1789761234.827419:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761234.827421:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761234.827425:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761234.827434:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496015104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761234.827441:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761234.827443:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761234.827445:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761234.827447:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1214 00000100:00080000:1.0:1789761234.827450:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761234.827453:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761234.827456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761234.827463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496015232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761234.827478:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761234.827517:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761234/real 1789761234] req@ffff8dd20504ce00 x1876699496014848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761234.827541:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496014848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761234.827551:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761234.827554:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761234.827557:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496014848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761234.827580:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496014976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761234.827597:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761234.827606:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761234/real 1789761234] req@ffff8dd20504f100 x1876699496014976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761234.827615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496014976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761234.827621:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761234.827623:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761234.827625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496014976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761234.827629:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496015104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761234.827641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761234.827646:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761234/real 1789761234] req@ffff8dd20504ed80 x1876699496015104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761234.827661:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496015104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761234.827666:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761234.827669:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761234.827671:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496015104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761234.827675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496015232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761234.827687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761234.827693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761234/real 1789761234] req@ffff8dd20504d880 x1876699496015232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761234.827701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496015232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761234.827706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761234.827708:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761234.827709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496015232:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761235.403332:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761236.427286:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761237.451317:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761238.475371:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761239.499320:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761239.947289:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761239.947302:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761239.947307:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761239.947319:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1219 00000100:00080000:1.0:1789761239.947330:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761239.947338:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761239.947382:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496015360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761239.947415:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761239.947420:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761239.947422:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761239.947427:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1219 00000100:00080000:1.0:1789761239.947433:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761239.947438:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761239.947448:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761239.947450:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496015360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761239.947471:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496015488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761239.947486:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761239.947490:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761239.947492:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761239.947495:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1219 00000100:00080000:1.0:1789761239.947499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761239.947503:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761239.947508:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789761239.947516:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761239.947521:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496015616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761239.947531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761239.947534:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761239.947536:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761239.947539:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1219 00000100:00080000:1.0:1789761239.947548:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761239.947551:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761239.947555:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761239.947566:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761239/real 1789761239] req@ffff8dd20504c000 x1876699496015360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761239.947567:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496015744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761239.947607:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496015360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761239.947617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761239.947621:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761239.947624:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496015360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761239.947645:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496015488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761239.947666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761239.947676:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761239/real 1789761239] req@ffff8dd20504ea00 x1876699496015488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761239.947691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496015488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761239.947701:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761239.947705:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761239.947709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496015488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761239.947716:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496015616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761239.947739:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761239.947750:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761239/real 1789761239] req@ffff8dd20504c380 x1876699496015616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761239.947759:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496015616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761239.947765:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761239.947768:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761239.947770:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496015616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761239.947774:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496015744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761239.947787:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761239.947794:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761239/real 1789761239] req@ffff8dd20504dc00 x1876699496015744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761239.947802:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496015744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761239.947813:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761239.947815:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761239.947817:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496015744:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761240.523332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761241.547353:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761242.571355:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761243.595346:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761244.619361:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761245.067256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761245.067267:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761245.067272:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761245.067307:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1224 00000100:00080000:1.0:1789761245.067328:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761245.067335:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761245.067367:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496015872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761245.067390:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761245.067393:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761245.067395:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761245.067398:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1224 00000100:00080000:1.0:1789761245.067400:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761245.067403:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761245.067411:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761245.067425:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496015872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761245.067429:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496016000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761245.067443:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761245.067448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761245.067450:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761245.067454:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1224 00000100:00080000:1.0:1789761245.067457:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761245.067460:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761245.067464:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761245.067474:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496016128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761245.067482:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761245.067485:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761245.067486:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761245.067489:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1224 00000100:00080000:1.0:1789761245.067491:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761245.067494:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1789761245.067500:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789761245.067501:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761245.067512:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496016256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761245.067548:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761245/real 1789761245] req@ffff8dd20504ca80 x1876699496015872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761245.067569:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496015872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761245.067576:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761245.067580:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761245.067583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496015872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761245.067601:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496016000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761245.067620:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761245.067629:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761245/real 1789761245] req@ffff8dd20504dc00 x1876699496016000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761245.067637:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496016000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761245.067643:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761245.067645:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761245.067647:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496016000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761245.067652:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496016128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761245.067663:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761245.067670:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761245/real 1789761245] req@ffff8dd20504c380 x1876699496016128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761245.067677:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496016128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761245.067682:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761245.067684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761245.067686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496016128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761245.067689:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496016256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761245.067699:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761245.067705:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761245/real 1789761245] req@ffff8dd20504ea00 x1876699496016256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761245.067712:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496016256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761245.067718:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761245.067719:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761245.067721:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496016256:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761245.643341:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761246.667379:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761247.691300:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761248.715348:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761249.739357:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761250.187307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761250.187319:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761250.187324:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761250.187335:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1229 00000100:00080000:1.0:1789761250.187382:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761250.187391:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761250.187428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496016384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761250.187462:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761250.187467:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761250.187469:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761250.187473:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1229 00000100:00080000:1.0:1789761250.187477:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761250.187482:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761250.187492:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761250.187506:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496016512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761250.187516:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496016384:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761250.187526:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761250.187532:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761250.187534:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761250.187541:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1229 00000100:00080000:1.0:1789761250.187547:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761250.187551:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761250.187558:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761250.187572:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496016640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761250.187584:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761250.187588:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761250.187590:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761250.187593:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1229 00000100:00080000:1.0:1789761250.187598:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761250.187602:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1789761250.187611:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789761250.187613:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761250.187628:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496016768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761250.187678:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761250/real 1789761250] req@ffff8dd20504c000 x1876699496016384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761250.187700:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496016384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761250.187712:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761250.187717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761250.187721:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496016384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761250.187746:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496016512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761250.187768:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761250.187778:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761250/real 1789761250] req@ffff8dd20504d880 x1876699496016512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761250.187787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496016512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761250.187795:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761250.187797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761250.187800:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496016512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761250.187821:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496016640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761250.187843:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761250.187856:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761250/real 1789761250] req@ffff8dd20504ed80 x1876699496016640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761250.187869:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496016640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761250.187885:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761250.187890:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761250.187893:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496016640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761250.187898:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496016768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761250.187915:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761250.187924:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761250/real 1789761250] req@ffff8dd20504f100 x1876699496016768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761250.187932:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496016768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761250.187937:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761250.187940:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761250.187942:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496016768:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761250.763356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761251.787334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761252.811351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761253.835365:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761254.859362:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761255.307298:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761255.307308:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761255.307311:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761255.307325:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1234 00000100:00080000:2.0:1789761255.307334:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761255.307340:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761255.307416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496016896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761255.307440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761255.307443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761255.307445:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761255.307448:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1234 00000100:00080000:2.0:1789761255.307451:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761255.307453:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761255.307460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761255.307470:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496017024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761255.307477:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761255.307480:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761255.307483:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761255.307486:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496016896:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761255.307488:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1234 00000100:00080000:2.0:1789761255.307494:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761255.307497:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761255.307500:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761255.307510:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496017152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761255.307517:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761255.307520:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761255.307522:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761255.307524:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1234 00000100:00080000:2.0:1789761255.307536:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761255.307539:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761255.307542:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761255.307550:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761255.307554:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496017280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761255.307592:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761255/real 1789761255] req@ffff8dd205264700 x1876699496016896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761255.307606:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496016896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761255.307613:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761255.307617:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761255.307620:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496016896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761255.307638:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496017024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761255.307653:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761255.307661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761255/real 1789761255] req@ffff8dd205266a00 x1876699496017024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761255.307669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496017024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761255.307675:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761255.307678:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761255.307681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496017024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761255.307687:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496017152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761255.307706:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761255.307718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761255/real 1789761255] req@ffff8dd205265880 x1876699496017152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761255.307731:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496017152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761255.307740:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761255.307742:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761255.307745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496017152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761255.307749:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496017280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761255.307779:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761255.307788:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761255/real 1789761255] req@ffff8dd205264000 x1876699496017280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761255.307797:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496017280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761255.307803:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761255.307805:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761255.307807:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496017280:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761255.883329:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761256.907425:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761257.931367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761258.955336:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761259.979365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761260.427244:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761260.427254:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761260.427257:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761260.427266:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1240 00000100:00080000:2.0:1789761260.427276:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761260.427289:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761260.427319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496017408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761260.427341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761260.427344:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761260.427345:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761260.427349:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1240 00000100:00080000:2.0:1789761260.427352:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761260.427355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761260.427362:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761260.427372:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496017408:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761260.427374:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496017536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761260.427393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761260.427395:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761260.427397:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761260.427400:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1240 00000100:00080000:2.0:1789761260.427402:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761260.427405:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761260.427408:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761260.427416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496017664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761260.427424:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761260.427426:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761260.427427:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761260.427432:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1240 00000800:00000100:3.0:1789761260.427433:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761260.427436:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761260.427439:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761260.427443:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761260.427461:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496017792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761260.427486:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761260/real 1789761260] req@ffff8dd205267b80 x1876699496017408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761260.427499:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496017408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761260.427508:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761260.427511:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761260.427515:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496017408:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761260.427534:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496017536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761260.427551:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761260.427561:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761260/real 1789761260] req@ffff8dd205264e00 x1876699496017536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761260.427569:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496017536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761260.427579:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761260.427581:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761260.427583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496017536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761260.427587:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496017664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761260.427597:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761260.427603:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761260/real 1789761260] req@ffff8dd205264380 x1876699496017664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761260.427610:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496017664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761260.427615:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761260.427617:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761260.427619:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496017664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761260.427622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496017792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761260.427632:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761260.427637:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761260/real 1789761260] req@ffff8dd205267800 x1876699496017792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761260.427645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496017792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761260.427649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761260.427651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761260.427654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496017792:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761261.003329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761262.027372:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761263.051294:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761264.075364:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761265.099314:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761265.547241:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761265.547254:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761265.547258:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761265.547267:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1245 00000100:00080000:2.0:1789761265.547277:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761265.547283:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761265.547313:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496017920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761265.547338:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761265.547341:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761265.547342:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761265.547345:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1245 00000100:00080000:2.0:1789761265.547347:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761265.547349:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761265.547356:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761265.547364:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496018048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761265.547371:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761265.547374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761265.547375:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761265.547378:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1245 00000100:00080000:2.0:1789761265.547381:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761265.547383:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761265.547387:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761265.547392:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496017920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761265.547398:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496018176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761265.547406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761265.547408:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761265.547410:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761265.547412:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1245 00000100:00080000:2.0:1789761265.547414:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761265.547416:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761265.547421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761265.547432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496018304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761265.547453:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761265.547500:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761265/real 1789761265] req@ffff8dd205267100 x1876699496017920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761265.547520:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496017920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761265.547531:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761265.547536:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761265.547540:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496017920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761265.547548:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496018048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761265.547570:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761265.547581:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761265/real 1789761265] req@ffff8dd205266680 x1876699496018048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761265.547589:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496018048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761265.547594:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761265.547597:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761265.547599:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496018048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761265.547619:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496018176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761265.547631:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761265.547638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761265/real 1789761265] req@ffff8dd205265500 x1876699496018176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761265.547646:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496018176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761265.547651:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761265.547653:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761265.547655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496018176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761265.547658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496018304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761265.547668:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761265.547674:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761265/real 1789761265] req@ffff8dd205264a80 x1876699496018304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761265.547682:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496018304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761265.547701:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761265.547705:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761265.547708:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496018304:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761266.123356:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761267.147314:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761268.171324:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761269.195327:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761270.219342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761270.667279:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761270.667294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761270.667298:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761270.667306:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1250 00000100:00080000:1.0:1789761270.667327:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761270.667334:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761270.667366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496018432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761270.667400:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761270.667403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761270.667404:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761270.667407:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1250 00000100:00080000:1.0:1789761270.667410:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761270.667412:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761270.667420:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761270.667429:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496018560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761270.667437:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761270.667453:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761270.667455:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761270.667456:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496018432:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761270.667460:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1250 00000100:00080000:1.0:1789761270.667465:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761270.667468:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761270.667471:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761270.667481:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496018688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761270.667488:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761270.667491:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761270.667492:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761270.667494:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1250 00000100:00080000:1.0:1789761270.667497:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761270.667499:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761270.667502:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761270.667518:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496018816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761270.667530:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761270.667569:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761270/real 1789761270] req@ffff8dd20504ce00 x1876699496018432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761270.667594:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496018432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761270.667602:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761270.667606:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761270.667610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496018432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761270.667635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496018560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761270.667687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761270.667702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761270/real 1789761270] req@ffff8dd20504f800 x1876699496018560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761270.667717:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496018560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761270.667726:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761270.667730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761270.667732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496018560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761270.667737:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496018688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761270.667758:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761270.667771:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761270/real 1789761270] req@ffff8dd20504fb80 x1876699496018688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761270.667784:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496018688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761270.667793:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761270.667797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761270.667800:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496018688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761270.667807:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496018816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761270.667824:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761270.667832:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761270/real 1789761270] req@ffff8dd20504df80 x1876699496018816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761270.667840:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496018816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761270.667845:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761270.667847:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761270.667849:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496018816:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761271.243358:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761272.267334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761273.291363:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761274.315315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761275.339313:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761275.787286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761275.787296:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761275.787300:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761275.787310:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1255 00000100:00080000:1.0:1789761275.787320:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761275.787326:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761275.787362:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496018944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761275.787386:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761275.787389:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761275.787390:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761275.787397:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1255 00000100:00080000:1.0:1789761275.787400:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761275.787402:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761275.787417:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761275.787437:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496018944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761275.787438:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496019072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761275.787455:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761275.787460:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761275.787462:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761275.787467:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1255 00000100:00080000:1.0:1789761275.787471:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761275.787476:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761275.787481:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761275.787492:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496019200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761275.787500:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761275.787503:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761275.787507:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761275.787509:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761275.787513:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1255 00000100:00080000:1.0:1789761275.787518:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761275.787521:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761275.787524:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761275.787533:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496019328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761275.787542:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761275/real 1789761275] req@ffff8dd20504d500 x1876699496018944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761275.787557:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496018944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761275.787567:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761275.787572:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761275.787576:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496018944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761275.787600:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496019072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761275.787640:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761275.787652:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761275/real 1789761275] req@ffff8dd20504c700 x1876699496019072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761275.787661:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496019072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761275.787668:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761275.787670:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761275.787672:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496019072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761275.787677:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496019200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761275.787691:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761275.787697:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761275/real 1789761275] req@ffff8dd20504e680 x1876699496019200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761275.787705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496019200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761275.787711:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761275.787713:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761275.787715:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496019200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761275.787718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496019328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761275.787729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761275.787735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761275/real 1789761275] req@ffff8dd20504d180 x1876699496019328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761275.787743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496019328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761275.787753:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761275.787755:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761275.787757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496019328:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761276.363349:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761277.387364:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761278.411321:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761279.435302:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761280.459307:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761280.907261:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761280.907276:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761280.907280:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761280.907290:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1260 00000100:00080000:1.0:1789761280.907300:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761280.907308:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761280.907342:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496019456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761280.907366:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761280.907369:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761280.907370:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761280.907374:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1260 00000100:00080000:1.0:1789761280.907377:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761280.907380:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761280.907388:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761280.907399:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496019584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761280.907407:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761280.907410:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761280.907412:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761280.907417:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1260 00000100:00100000:0.0:1789761280.907420:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496019456:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761280.907420:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761280.907423:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761280.907431:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761280.907439:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496019712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761280.907446:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761280.907448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761280.907450:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761280.907452:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1260 00000100:00080000:1.0:1789761280.907454:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761280.907456:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761280.907459:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761280.907467:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496019840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761280.907530:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496019584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761280.907553:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496019712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761280.907584:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496019840:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761281.483350:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761282.507307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761283.531268:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761283.979439:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789761283.979499:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761283.979513:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761283.979549:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761283.979573:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761283.979584:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789761283.979595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761280/real 1789761283] req@ffff8dd20504d180 x1876699496019584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761283.979622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496019584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761283.979636:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761283.979641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761283.979646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496019584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761283.979655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761280/real 1789761283] req@ffff8dd20504e680 x1876699496019712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761283.979669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496019712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761283.979679:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761283.979682:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761283.979686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496019712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761283.979693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761280/real 1789761283] req@ffff8dd20504c700 x1876699496019840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761283.979706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496019840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761283.979716:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761283.979720:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761283.979724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496019840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761283.979731:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761280/real 1789761283] req@ffff8dd20504e300 x1876699496019456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761283.979744:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496019456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761283.979752:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761283.979757:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761283.979761:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496019456:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789761284.555326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761285.579357:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761286.027306:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761286.027319:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761286.027324:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761286.027334:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1265 00000100:00080000:2.0:1789761286.027346:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761286.027353:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761286.027388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496019968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761286.027426:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761286.027430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761286.027432:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761286.027436:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1265 00000100:00080000:2.0:1789761286.027441:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761286.027445:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761286.027455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761286.027465:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496019968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761286.027474:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496020096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761286.027493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761286.027497:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761286.027500:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761286.027504:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1265 00000100:00080000:2.0:1789761286.027516:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761286.027521:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761286.027528:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789761286.027533:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761286.027562:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496020224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761286.027575:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761286.027579:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761286.027582:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761286.027587:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1265 00000100:00080000:2.0:1789761286.027591:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761286.027596:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761286/real 1789761286] req@ffff8dd205267480 x1876699496019968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761286.027599:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761286.027608:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761286.027624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496019968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761286.027626:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496020352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761286.027654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761286.027662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761286.027667:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496019968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761286.027693:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496020096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761286.027723:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761286.027737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761286/real 1789761286] req@ffff8dd205264a80 x1876699496020096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761286.027751:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496020096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761286.027761:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761286.027764:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761286.027766:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496020096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789761286.027772:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496020224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789761286.027794:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761286.027806:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761286/real 1789761286] req@ffff8dd205265500 x1876699496020224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761286.027817:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496020224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761286.027826:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761286.027829:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761286.027832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496020224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789761286.027838:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496020352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789761286.027858:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761286.027869:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761286/real 1789761286] req@ffff8dd205266680 x1876699496020352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761286.027888:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496020352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761286.027897:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761286.027900:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789761286.027904:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496020352:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761286.603308:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761287.627357:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761288.651346:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761289.675423:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761290.699343:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761291.147257:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761291.147267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761291.147271:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761291.147280:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1270 00000100:00080000:2.0:1789761291.147290:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761291.147296:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761291.147330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496020480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761291.147355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761291.147358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761291.147360:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761291.147362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1270 00000100:00080000:2.0:1789761291.147365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761291.147368:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761291.147375:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761291.147385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496020608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761291.147394:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496020480:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761291.147395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761291.147403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761291.147405:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761291.147408:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1270 00000100:00080000:2.0:1789761291.147412:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761291.147416:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761291.147422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761291.147436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496020736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761291.147446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761291.147450:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761291.147452:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761291.147457:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761291.147461:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1270 00000100:00080000:2.0:1789761291.147465:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761291.147469:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761291.147473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761291.147483:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496020864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761291.147510:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761291/real 1789761291] req@ffff8dd205267100 x1876699496020480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761291.147526:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496020480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761291.147534:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761291.147544:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761291.147549:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496020480:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761291.147574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496020608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761291.147599:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761291.147610:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761291/real 1789761291] req@ffff8dd205267800 x1876699496020608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761291.147620:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496020608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761291.147626:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761291.147628:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761291.147631:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496020608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761291.147635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496020736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761291.147648:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761291.147655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761291/real 1789761291] req@ffff8dd205264380 x1876699496020736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761291.147662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496020736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761291.147668:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761291.147669:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761291.147671:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496020736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761291.147675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496020864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761291.147686:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761291.147692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761291/real 1789761291] req@ffff8dd205264e00 x1876699496020864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761291.147699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496020864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761291.147705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761291.147706:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761291.147708:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496020864:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761291.723331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761292.747373:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761293.771373:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761294.795314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761295.819345:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761296.267294:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761296.267303:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761296.267307:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761296.267316:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1275 00000100:00080000:1.0:1789761296.267324:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761296.267336:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761296.267372:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496020992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761296.267394:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761296.267397:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761296.267398:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761296.267402:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1275 00000100:00080000:1.0:1789761296.267404:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761296.267406:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761296.267414:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761296.267423:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496021120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761296.267431:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761296.267434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761296.267435:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761296.267438:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1275 00000100:00080000:1.0:1789761296.267441:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761296.267443:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496020992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761296.267447:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761296.267454:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761296.267480:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496021248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761296.267492:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761296.267495:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761296.267498:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761296.267502:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1275 00000100:00080000:1.0:1789761296.267516:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761296.267521:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761296.267526:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789761296.267528:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761296.267539:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496021376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761296.267571:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761296/real 1789761296] req@ffff8dd20504d500 x1876699496020992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761296.267585:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496020992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761296.267592:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761296.267595:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761296.267598:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496020992:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761296.267623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496021120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761296.267640:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761296.267649:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761296/real 1789761296] req@ffff8dd20504df80 x1876699496021120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761296.267661:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496021120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761296.267667:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761296.267669:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761296.267671:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496021120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761296.267675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496021248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761296.267686:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761296.267692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761296/real 1789761296] req@ffff8dd20504fb80 x1876699496021248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761296.267699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496021248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761296.267704:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761296.267706:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761296.267707:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496021248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761296.267711:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496021376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761296.267721:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761296.267726:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761296/real 1789761296] req@ffff8dd20504f800 x1876699496021376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761296.267733:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496021376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761296.267742:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761296.267744:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761296.267746:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496021376:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761296.843338:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761297.867334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761298.891346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761299.915340:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761300.939339:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761301.387285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761301.387295:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761301.387298:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761301.387307:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1280 00000100:00080000:0.0:1789761301.387316:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761301.387323:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761301.387362:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496021504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761301.387387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761301.387390:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761301.387391:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761301.387394:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1280 00000100:00080000:0.0:1789761301.387397:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761301.387401:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761301.387409:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761301.387418:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496021504:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761301.387431:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496021632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761301.387446:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761301.387450:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761301.387452:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761301.387458:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1280 00000100:00080000:0.0:1789761301.387462:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761301.387467:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761301.387473:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761301.387487:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496021760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761301.387495:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761301.387499:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761301.387502:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761301.387504:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761301.387508:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1280 00000100:00080000:0.0:1789761301.387512:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761301.387515:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761301.387520:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761301.387528:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496021888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761301.387546:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761301/real 1789761301] req@ffff8dd211834a80 x1876699496021504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761301.387562:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496021504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761301.387570:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761301.387573:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761301.387576:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496021504:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761301.387595:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496021632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761301.387612:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761301.387621:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761301/real 1789761301] req@ffff8dd211836680 x1876699496021632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761301.387629:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496021632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761301.387635:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761301.387638:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761301.387640:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496021632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761301.387644:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496021760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761301.387655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761301.387660:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761301/real 1789761301] req@ffff8dd211834380 x1876699496021760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761301.387668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496021760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761301.387673:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761301.387675:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761301.387677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496021760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761301.387680:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496021888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761301.387690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761301.387695:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761301/real 1789761301] req@ffff8dd211835f80 x1876699496021888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761301.387703:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496021888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761301.387715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761301.387717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761301.387719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496021888:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761301.963303:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761302.987346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761304.011326:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761305.035370:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761306.059315:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761306.507271:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761306.507281:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761306.507285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761306.507295:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1286 00000100:00080000:0.0:1789761306.507308:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761306.507317:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761306.507361:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496022016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761306.507402:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761306.507405:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761306.507406:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761306.507410:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1286 00000100:00080000:0.0:1789761306.507412:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761306.507415:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761306.507426:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761306.507438:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496022144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761306.507446:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496022016:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761306.507448:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761306.507455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761306.507457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761306.507460:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1286 00000100:00080000:0.0:1789761306.507463:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761306.507465:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761306.507469:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761306.507478:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496022272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761306.507484:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761306.507487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761306.507488:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761306.507491:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1286 00000100:00080000:0.0:1789761306.507512:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761306.507515:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761306.507520:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789761306.507521:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761306.507532:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496022400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761306.507563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761306/real 1789761306] req@ffff8dd211834700 x1876699496022016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761306.507578:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496022016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761306.507587:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761306.507590:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761306.507594:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496022016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761306.507615:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496022144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761306.507630:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761306.507639:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761306/real 1789761306] req@ffff8dd211834000 x1876699496022144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761306.507647:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496022144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761306.507653:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761306.507655:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761306.507657:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496022144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761306.507661:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496022272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761306.507672:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761306.507678:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761306/real 1789761306] req@ffff8dd211836a00 x1876699496022272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761306.507691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496022272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761306.507696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761306.507698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761306.507700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496022272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761306.507703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496022400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761306.507713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761306.507719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761306/real 1789761306] req@ffff8dd211835c00 x1876699496022400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761306.507727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496022400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761306.507732:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761306.507734:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761306.507735:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496022400:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761307.083421:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761308.107338:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761309.131348:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761310.155371:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761311.179336:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761311.627303:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761311.627313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761311.627317:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761311.627327:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1291 00000100:00080000:1.0:1789761311.627341:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761311.627359:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761311.627411:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496022528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761311.627438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761311.627441:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761311.627442:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761311.627446:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1291 00000100:00080000:1.0:1789761311.627449:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761311.627452:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761311.627463:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761311.627477:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496022656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761311.627489:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761311.627494:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789761311.627495:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496022528:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761311.627496:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761311.627503:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1291 00000100:00080000:1.0:1789761311.627507:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761311.627510:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761311.627514:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761311.627523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496022784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761311.627531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761311.627533:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761311.627535:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761311.627537:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1291 00000100:00080000:1.0:1789761311.627540:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761311.627542:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761311.627546:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761311.627554:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496022912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761311.627581:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761311.627631:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761311/real 1789761311] req@ffff8dd20504ce00 x1876699496022528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761311.627652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496022528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761311.627666:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761311.627670:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761311.627675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496022528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761311.627704:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496022656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761311.627731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761311.627743:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761311/real 1789761311] req@ffff8dd20504f100 x1876699496022656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761311.627752:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496022656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761311.627758:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761311.627760:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761311.627763:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496022656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761311.627768:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496022784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761311.627781:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761311.627788:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761311/real 1789761311] req@ffff8dd20504ed80 x1876699496022784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761311.627796:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496022784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761311.627801:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761311.627803:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761311.627805:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496022784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761311.627808:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496022912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761311.627818:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761311.627824:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761311/real 1789761311] req@ffff8dd20504d880 x1876699496022912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761311.627832:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496022912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761311.627841:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761311.627844:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761311.627848:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496022912:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761312.203405:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761313.227346:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761314.251365:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761315.275331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761316.299348:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761316.747337:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761316.747346:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761316.747351:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761316.747362:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1296 00000100:00080000:0.0:1789761316.747375:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761316.747382:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761316.747427:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496023040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761316.747455:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761316.747458:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761316.747459:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761316.747462:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1296 00000100:00080000:0.0:1789761316.747466:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761316.747469:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761316.747480:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761316.747491:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496023168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761316.747500:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761316.747502:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761316.747504:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761316.747507:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1296 00000100:00080000:0.0:1789761316.747510:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761316.747518:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761316.747520:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496023040:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789761316.747523:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761316.747540:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496023296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761316.747553:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761316.747555:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761316.747557:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761316.747560:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1296 00000100:00080000:0.0:1789761316.747563:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761316.747567:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761316.747571:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761316.747580:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496023424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761316.747604:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761316.747648:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761316/real 1789761316] req@ffff8dd211837b80 x1876699496023040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761316.747662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496023040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761316.747671:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761316.747674:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761316.747677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496023040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761316.747698:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496023168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761316.747714:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761316.747722:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761316/real 1789761316] req@ffff8dd211837100 x1876699496023168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761316.747730:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496023168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761316.747736:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761316.747738:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761316.747741:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496023168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761316.747745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496023296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761316.747757:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761316.747762:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761316/real 1789761316] req@ffff8dd211836300 x1876699496023296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761316.747769:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496023296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761316.747775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761316.747777:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761316.747779:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496023296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761316.747784:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496023424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761316.747803:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761316.747815:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761316/real 1789761316] req@ffff8dd211837800 x1876699496023424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761316.747828:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496023424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761316.747837:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761316.747840:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761316.747843:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496023424:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761317.323348:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761318.347375:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761319.371424:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761320.395378:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761321.419322:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761321.867290:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761321.867299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761321.867303:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761321.867315:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1301 00000100:00080000:2.0:1789761321.867326:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761321.867334:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761321.867372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496023552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761321.867396:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761321.867399:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761321.867401:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761321.867404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1301 00000100:00080000:2.0:1789761321.867407:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761321.867411:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761321.867419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761321.867431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496023680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761321.867433:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496023552:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761321.867447:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761321.867449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761321.867450:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761321.867453:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1301 00000100:00080000:2.0:1789761321.867455:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761321.867457:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761321.867461:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761321.867470:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496023808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761321.867478:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761321.867480:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761321.867481:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761321.867483:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1301 00000100:00080000:2.0:1789761321.867486:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761321.867488:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761321.867491:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761321.867496:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761321.867500:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496023936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761321.867536:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761321/real 1789761321] req@ffff8dd205267b80 x1876699496023552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761321.867549:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496023552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761321.867557:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761321.867560:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761321.867565:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496023552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761321.867588:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496023680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761321.867613:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761321.867626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761321/real 1789761321] req@ffff8dd205264000 x1876699496023680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761321.867635:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496023680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761321.867642:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761321.867644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761321.867648:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496023680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761321.867655:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496023808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761321.867675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761321.867688:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761321/real 1789761321] req@ffff8dd205265880 x1876699496023808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761321.867701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496023808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761321.867716:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761321.867720:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761321.867724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496023808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761321.867731:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496023936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761321.867752:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761321.867764:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761321/real 1789761321] req@ffff8dd205266a00 x1876699496023936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761321.867776:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496023936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761321.867785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761321.867788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761321.867792:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496023936:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761322.443367:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761323.467332:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761324.491359:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761325.515342:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761326.539343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761326.987299:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761326.987308:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761326.987312:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761326.987323:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1306 00000100:00080000:0.0:1789761326.987335:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761326.987341:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761326.987389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496024064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761326.987422:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761326.987427:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761326.987429:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761326.987435:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1306 00000100:00080000:0.0:1789761326.987441:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761326.987446:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761326.987459:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761326.987476:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496024192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761326.987483:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496024064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761326.987494:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761326.987500:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761326.987502:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761326.987508:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1306 00000100:00080000:0.0:1789761326.987513:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761326.987518:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761326.987524:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761326.987539:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496024320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761326.987560:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761326.987565:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761326.987567:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761326.987571:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761326.987576:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1306 00000100:00080000:0.0:1789761326.987581:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761326.987585:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761326.987590:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761326.987602:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496024448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761326.987623:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761326/real 1789761326] req@ffff8dd211837800 x1876699496024064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761326.987648:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496024064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761326.987660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761326.987666:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761326.987670:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496024064:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761326.987699:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496024192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761326.987724:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761326.987737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761326/real 1789761326] req@ffff8dd211836300 x1876699496024192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761326.987752:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496024192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761326.987761:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761326.987765:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761326.987769:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496024192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761326.987776:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496024320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761326.987794:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761326.987805:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761326/real 1789761326] req@ffff8dd211837100 x1876699496024320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761326.987818:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496024320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761326.987828:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761326.987832:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761326.987836:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496024320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761326.987842:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496024448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761326.987861:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761326.987872:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761326/real 1789761326] req@ffff8dd211837b80 x1876699496024448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761326.987891:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496024448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761326.987901:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761326.987904:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761326.987908:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496024448:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761327.563335:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761328.587386:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761329.611339:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761330.635370:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761331.659350:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761332.107305:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761332.107315:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761332.107319:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761332.107328:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1311 00000100:00080000:0.0:1789761332.107337:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761332.107343:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761332.107376:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496024576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761332.107400:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761332.107403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761332.107405:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761332.107407:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1311 00000100:00080000:0.0:1789761332.107410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761332.107412:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761332.107419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761332.107427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496024704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761332.107434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761332.107437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761332.107439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761332.107441:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1311 00000100:00080000:0.0:1789761332.107444:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761332.107447:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761332.107451:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761332.107457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496024832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761332.107465:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496024576:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761332.107467:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761332.107471:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761332.107475:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761332.107479:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1311 00000100:00080000:0.0:1789761332.107481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761332.107483:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761332.107486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761332.107494:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496024960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761332.107534:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761332.107576:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761332/real 1789761332] req@ffff8dd211835c00 x1876699496024576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761332.107594:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496024576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761332.107605:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761332.107609:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761332.107614:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496024576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761332.107622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496024704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761332.107643:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761332.107655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761332/real 1789761332] req@ffff8dd211836a00 x1876699496024704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761332.107668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496024704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761332.107685:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761332.107688:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761332.107692:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496024704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761332.107718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496024832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761332.107738:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761332.107749:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761332/real 1789761332] req@ffff8dd211834000 x1876699496024832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761332.107761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496024832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761332.107771:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761332.107775:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761332.107778:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496024832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761332.107785:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496024960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761332.107801:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761332.107812:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761332/real 1789761332] req@ffff8dd211834700 x1876699496024960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761332.107825:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496024960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761332.107834:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761332.107837:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761332.107841:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496024960:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761332.683332:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761333.707366:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761334.731331:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761335.755351:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761336.779322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761337.227301:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761337.227313:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761337.227316:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761337.227325:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1316 00000100:00080000:2.0:1789761337.227336:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761337.227342:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761337.227392:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496025088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761337.227417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761337.227420:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761337.227422:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761337.227425:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1316 00000100:00080000:2.0:1789761337.227427:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761337.227430:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761337.227438:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761337.227467:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496025216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761337.227479:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761337.227484:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789761337.227485:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496025088:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761337.227487:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761337.227496:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1316 00000100:00080000:2.0:1789761337.227500:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761337.227505:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761337.227512:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761337.227527:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496025344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761337.227539:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761337.227543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761337.227546:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761337.227549:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761337.227554:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1316 00000100:00080000:2.0:1789761337.227560:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761337.227565:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761337.227570:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761337.227581:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496025472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761337.227601:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761337/real 1789761337] req@ffff8dd205264700 x1876699496025088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761337.227622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496025088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761337.227644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761337.227650:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761337.227654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496025088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761337.227678:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496025216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761337.227701:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761337.227715:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761337/real 1789761337] req@ffff8dd205266300 x1876699496025216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761337.227728:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496025216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761337.227737:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761337.227740:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761337.227744:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496025216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761337.227750:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496025344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761337.227769:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761337.227780:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761337/real 1789761337] req@ffff8dd205265180 x1876699496025344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761337.227793:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496025344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761337.227802:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761337.227806:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761337.227809:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496025344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761337.227815:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496025472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761337.227833:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761337.227843:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761337/real 1789761337] req@ffff8dd205266d80 x1876699496025472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761337.227855:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496025472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761337.227862:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761337.227866:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761337.227870:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496025472:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761337.803328:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761338.827328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761339.851339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761340.875346:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761341.899330:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761342.347265:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761342.347274:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761342.347278:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761342.347288:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1321 00000100:00080000:2.0:1789761342.347298:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761342.347304:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761342.347339:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496025600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761342.347372:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761342.347374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761342.347376:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761342.347378:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1321 00000100:00080000:2.0:1789761342.347381:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761342.347383:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761342.347390:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761342.347400:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496025728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761342.347408:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761342.347410:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761342.347412:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761342.347414:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1321 00000100:00080000:2.0:1789761342.347416:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761342.347419:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761342.347422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761342.347428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496025856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761342.347438:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496025600:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761342.347438:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761342.347444:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761342.347448:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761342.347452:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1321 00000100:00080000:2.0:1789761342.347454:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761342.347456:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761342.347459:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761342.347471:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496025984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761342.347524:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761342.347563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761342/real 1789761342] req@ffff8dd205265f80 x1876699496025600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761342.347576:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496025600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761342.347584:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761342.347587:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761342.347590:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496025600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761342.347595:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496025728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761342.347611:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761342.347619:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761342/real 1789761342] req@ffff8dd205266d80 x1876699496025728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761342.347626:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496025728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761342.347632:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761342.347634:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761342.347636:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496025728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761342.347656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496025856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761342.347667:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761342.347673:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761342/real 1789761342] req@ffff8dd205265180 x1876699496025856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761342.347680:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496025856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761342.347686:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761342.347688:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761342.347690:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496025856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761342.347694:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496025984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761342.347703:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761342.347709:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761342/real 1789761342] req@ffff8dd205266300 x1876699496025984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761342.347716:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496025984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761342.347721:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761342.347723:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761342.347724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496025984:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761342.923340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761343.947336:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761344.971331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761345.995379:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761347.019296:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761347.467298:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761347.467311:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761347.467316:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761347.467328:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1327 00000100:00080000:1.0:1789761347.467343:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761347.467351:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761347.467389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496026112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761347.467417:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761347.467421:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761347.467423:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761347.467428:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1327 00000100:00080000:1.0:1789761347.467432:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761347.467437:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761347.467446:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761347.467460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496026240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761347.467471:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761347.467476:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761347.467478:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761347.467485:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1327 00000100:00100000:0.0:1789761347.467487:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496026112:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761347.467490:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761347.467498:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761347.467504:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761347.467517:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496026368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761347.467531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761347.467535:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761347.467537:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761347.467541:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1327 00000100:00080000:1.0:1789761347.467545:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761347.467549:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761347.467554:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761347.467566:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496026496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761347.467585:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496026240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761347.467616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496026368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761347.467637:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496026496:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761348.043319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761349.067342:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761350.091401:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761350.539488:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789761350.539550:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761350.539587:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761350.539656:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761350.539668:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761350.539678:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789761350.539811:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761347/real 1789761350] req@ffff8dd20504c380 x1876699496026368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761350.539839:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496026368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761350.539852:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761350.539858:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761350.539863:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496026368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761350.539871:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761347/real 1789761350] req@ffff8dd20504dc00 x1876699496026496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761350.539883:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496026496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761350.539891:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761350.539895:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761350.539898:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496026496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761350.539905:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761347/real 1789761350] req@ffff8dd20504c000 x1876699496026112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761350.539917:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496026112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761350.539926:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761350.539930:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761350.539933:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496026112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761350.539940:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761347/real 1789761350] req@ffff8dd20504ea00 x1876699496026240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761350.539953:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496026240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761350.539963:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761350.539966:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761350.539970:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496026240:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761351.115363:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761352.139326:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761352.587271:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761352.587282:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761352.587285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761352.587294:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1332 00000100:00080000:2.0:1789761352.587303:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761352.587309:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761352.587338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496026624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761352.587364:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761352.587367:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761352.587368:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761352.587371:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1332 00000100:00080000:2.0:1789761352.587373:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761352.587375:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761352.587383:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761352.587398:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496026752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761352.587409:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761352.587413:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761352.587414:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761352.587419:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1332 00000100:00080000:2.0:1789761352.587424:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761352.587429:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761352.587435:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761352.587444:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496026624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761352.587457:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496026880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761352.587471:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761352.587474:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761352.587476:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761352.587480:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1332 00000100:00080000:2.0:1789761352.587484:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761352.587488:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761352.587494:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761352.587512:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496027008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761352.587550:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761352.587602:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761352/real 1789761352] req@ffff8dd205264700 x1876699496026624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761352.587624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496026624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761352.587638:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761352.587644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761352.587649:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496026624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761352.587659:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496026752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761352.587695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761352.587712:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761352/real 1789761352] req@ffff8dd205266a00 x1876699496026752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761352.587726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496026752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761352.587741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761352.587746:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761352.587749:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496026752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761352.587785:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496026880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761352.587810:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761352.587823:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761352/real 1789761352] req@ffff8dd205265880 x1876699496026880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761352.587835:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496026880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761352.587844:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761352.587847:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761352.587851:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496026880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761352.587858:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496027008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761352.587879:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761352.587892:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761352/real 1789761352] req@ffff8dd205264000 x1876699496027008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761352.587905:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496027008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761352.587914:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761352.587917:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761352.587922:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496027008:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761353.163293:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761354.187335:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761355.211335:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761356.235309:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761357.259371:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761357.707270:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761357.707280:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761357.707284:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761357.707293:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1337 00000100:00080000:2.0:1789761357.707305:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761357.707313:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761357.707353:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496027136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761357.707377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761357.707380:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761357.707381:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761357.707384:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1337 00000100:00080000:2.0:1789761357.707387:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761357.707389:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761357.707399:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761357.707409:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496027264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761357.707419:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761357.707422:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761357.707423:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761357.707426:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1337 00000100:00080000:2.0:1789761357.707428:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761357.707432:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761357.707439:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496027136:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789761357.707440:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761357.707456:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496027392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761357.707467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761357.707470:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761357.707471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761357.707475:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1337 00000100:00080000:2.0:1789761357.707482:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761357.707485:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761357.707490:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761357.707498:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496027520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761357.707517:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761357.707560:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761357/real 1789761357] req@ffff8dd205267b80 x1876699496027136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761357.707576:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496027136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761357.707590:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761357.707593:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761357.707596:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496027136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761357.707603:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496027264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761357.707621:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761357.707630:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761357/real 1789761357] req@ffff8dd205264e00 x1876699496027264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761357.707638:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496027264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761357.707644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761357.707647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761357.707651:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496027264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761357.707678:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496027392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761357.707703:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761357.707715:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761357/real 1789761357] req@ffff8dd205264380 x1876699496027392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761357.707723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496027392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761357.707729:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761357.707731:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761357.707733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496027392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761357.707737:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496027520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761357.707751:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761357.707757:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761357/real 1789761357] req@ffff8dd205267800 x1876699496027520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761357.707764:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496027520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761357.707769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761357.707771:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761357.707773:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496027520:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761358.283266:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761359.307350:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761360.331307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761361.355321:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761362.379301:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761362.827242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761362.827251:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761362.827255:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761362.827266:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1342 00000100:00080000:2.0:1789761362.827279:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761362.827287:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761362.827329:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496027648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761362.827357:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761362.827360:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761362.827361:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761362.827365:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1342 00000100:00080000:2.0:1789761362.827367:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761362.827369:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761362.827378:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761362.827388:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496027776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761362.827396:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761362.827398:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761362.827400:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761362.827403:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1342 00000100:00080000:2.0:1789761362.827405:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761362.827407:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761362.827410:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761362.827418:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496027904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761362.827424:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496027648:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761362.827431:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761362.827434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761362.827435:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761362.827438:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1342 00000100:00080000:2.0:1789761362.827440:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761362.827442:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761362.827445:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761362.827453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496028032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761362.827505:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761362.827546:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761362/real 1789761362] req@ffff8dd205267100 x1876699496027648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761362.827567:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496027648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761362.827577:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761362.827580:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761362.827584:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496027648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761362.827590:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496027776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761362.827607:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761362.827616:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761362/real 1789761362] req@ffff8dd205266680 x1876699496027776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761362.827625:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496027776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761362.827631:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761362.827633:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761362.827636:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496027776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761362.827657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496027904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761362.827670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761362.827676:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761362/real 1789761362] req@ffff8dd205265500 x1876699496027904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761362.827684:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496027904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761362.827694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761362.827697:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761362.827699:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496027904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761362.827703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496028032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761362.827712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761362.827718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761362/real 1789761362] req@ffff8dd205264a80 x1876699496028032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761362.827725:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496028032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761362.827731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761362.827733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761362.827734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496028032:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761363.403459:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761364.427312:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761365.451356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761366.475262:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761367.499339:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761367.947316:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761367.947326:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761367.947330:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761367.947340:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1347 00000100:00080000:0.0:1789761367.947353:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761367.947361:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761367.947403:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496028160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761367.947432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761367.947435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761367.947437:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761367.947440:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1347 00000100:00080000:0.0:1789761367.947443:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761367.947445:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761367.947455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761367.947465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496028288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761367.947473:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761367.947476:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761367.947479:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761367.947480:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496028160:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761367.947483:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1347 00000100:00080000:0.0:1789761367.947487:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761367.947493:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761367.947498:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761367.947514:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496028416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761367.947525:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761367.947530:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761367.947532:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761367.947536:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1347 00000100:00080000:0.0:1789761367.947543:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761367.947548:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761367.947554:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789761367.947555:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761367.947568:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496028544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761367.947599:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761367/real 1789761367] req@ffff8dd211835f80 x1876699496028160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761367.947616:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496028160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761367.947625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761367.947628:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761367.947631:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496028160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761367.947654:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496028288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761367.947670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761367.947679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761367/real 1789761367] req@ffff8dd211834380 x1876699496028288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761367.947687:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496028288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761367.947693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761367.947696:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761367.947698:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496028288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761367.947701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496028416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761367.947713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761367.947719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761367/real 1789761367] req@ffff8dd211836680 x1876699496028416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761367.947732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496028416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761367.947737:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761367.947739:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761367.947741:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496028416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761367.947745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496028544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761367.947756:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761367.947762:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761367/real 1789761367] req@ffff8dd211834a80 x1876699496028544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761367.947770:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496028544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761367.947775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761367.947776:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761367.947778:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496028544:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761368.523352:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761369.547362:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761370.571383:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761371.595378:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761372.619357:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761373.067323:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761373.067333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761373.067338:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761373.067350:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1352 00000100:00080000:2.0:1789761373.067363:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761373.067370:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761373.067416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496028672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761373.067450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761373.067454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761373.067455:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761373.067459:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1352 00000100:00080000:2.0:1789761373.067462:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761373.067466:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761373.067476:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761373.067497:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496028672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761373.067500:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496028800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761373.067512:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761373.067515:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761373.067518:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761373.067523:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1352 00000100:00080000:2.0:1789761373.067527:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761373.067531:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761373.067538:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761373.067553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496028928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761373.067565:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761373.067569:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789761373.067572:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761373.067573:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761373.067580:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1352 00000100:00080000:2.0:1789761373.067585:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761373.067589:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761373.067594:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761373.067605:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496029056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761373.067622:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761373/real 1789761373] req@ffff8dd205267480 x1876699496028672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761373.067644:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496028672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761373.067663:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761373.067668:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761373.067672:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496028672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761373.067702:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496028800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761373.067731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761373.067746:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761373/real 1789761373] req@ffff8dd205264a80 x1876699496028800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761373.067761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496028800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761373.067771:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761373.067774:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761373.067778:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496028800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761373.067785:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496028928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761373.067809:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761373.067824:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761373/real 1789761373] req@ffff8dd205265500 x1876699496028928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761373.067847:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496028928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761373.067857:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761373.067861:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761373.067863:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496028928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761373.067869:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496029056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761373.067903:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761373.067912:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761373/real 1789761373] req@ffff8dd205266680 x1876699496029056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761373.067921:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496029056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761373.067926:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761373.067928:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761373.067930:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496029056:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761373.643298:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761374.667358:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761375.691355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761376.715357:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761377.739330:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761378.187260:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761378.187271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761378.187277:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761378.187289:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1357 00000100:00080000:2.0:1789761378.187303:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761378.187312:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761378.187354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496029184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761378.187387:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761378.187391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761378.187393:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761378.187398:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1357 00000100:00080000:2.0:1789761378.187403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761378.187409:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761378.187422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761378.187439:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496029184:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761378.187441:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496029312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761378.187458:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761378.187462:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761378.187465:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761378.187470:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1357 00000100:00080000:2.0:1789761378.187473:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761378.187478:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761378.187484:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761378.187497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496029440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761378.187509:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761378.187512:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761378.187515:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761378.187519:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1357 00000100:00080000:2.0:1789761378.187523:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761378.187527:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761378.187532:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761378.187538:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761378.187548:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496029568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761378.187586:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761378/real 1789761378] req@ffff8dd205267100 x1876699496029184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761378.187608:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496029184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761378.187622:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761378.187627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761378.187632:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496029184:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761378.187661:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496029312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761378.187690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761378.187705:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761378/real 1789761378] req@ffff8dd205267800 x1876699496029312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761378.187719:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496029312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761378.187729:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761378.187732:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761378.187736:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496029312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761378.187742:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496029440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761378.187762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761378.187773:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761378/real 1789761378] req@ffff8dd205264380 x1876699496029440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761378.187787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496029440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761378.187795:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761378.187799:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761378.187802:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496029440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761378.187808:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496029568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761378.187830:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761378.187841:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761378/real 1789761378] req@ffff8dd205264e00 x1876699496029568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761378.187866:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496029568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761378.187882:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761378.187885:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761378.187889:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496029568:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761378.763385:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761379.787330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761380.811325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761381.835318:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761382.859366:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761383.307310:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761383.307321:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761383.307325:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761383.307336:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1362 00000100:00080000:1.0:1789761383.307348:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761383.307355:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761383.307398:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496029696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761383.307428:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761383.307431:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761383.307433:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761383.307436:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1362 00000100:00080000:1.0:1789761383.307439:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761383.307441:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761383.307451:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761383.307463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496029824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761383.307473:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761383.307475:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761383.307478:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761383.307484:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496029696:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761383.307488:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1362 00000100:00080000:1.0:1789761383.307499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761383.307507:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761383.307516:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761383.307535:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496029952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761383.307548:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761383.307553:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761383.307556:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761383.307562:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761383.307567:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1362 00000100:00080000:1.0:1789761383.307575:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761383.307581:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761383.307589:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761383.307604:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761383/real 1789761383] req@ffff8dd20504ca80 x1876699496029696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761383.307619:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496030080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761383.307631:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496029696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761383.307649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761383.307655:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761383.307663:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496029696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761383.307694:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496029824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761383.307718:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761383.307732:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761383/real 1789761383] req@ffff8dd20504ea00 x1876699496029824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761383.307747:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496029824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761383.307757:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761383.307760:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761383.307764:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496029824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761383.307772:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496029952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761383.307797:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761383.307806:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761383/real 1789761383] req@ffff8dd20504c000 x1876699496029952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761383.307827:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496029952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761383.307833:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761383.307836:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761383.307838:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496029952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761383.307843:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496030080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761383.307862:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761383.307874:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761383/real 1789761383] req@ffff8dd20504dc00 x1876699496030080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761383.307887:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496030080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761383.307896:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761383.307899:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761383.307902:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496030080:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761383.883312:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761384.907295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761385.931336:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761386.955392:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761387.979380:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761388.427345:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761388.427357:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761388.427362:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761388.427375:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1368 00000100:00080000:2.0:1789761388.427389:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761388.427396:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761388.427442:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496030208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761388.427470:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761388.427473:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761388.427475:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761388.427479:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1368 00000100:00080000:2.0:1789761388.427483:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761388.427487:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761388.427499:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761388.427516:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496030336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761388.427530:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496030208:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761388.427531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761388.427540:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761388.427543:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761388.427549:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1368 00000100:00080000:2.0:1789761388.427554:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761388.427559:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761388.427566:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761388.427582:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496030464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761388.427595:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761388.427600:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761388.427603:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761388.427604:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761388.427612:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1368 00000100:00080000:2.0:1789761388.427619:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761388.427624:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761388.427631:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761388.427647:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496030592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761388.427651:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761388/real 1789761388] req@ffff8dd205267b80 x1876699496030208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761388.427689:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496030208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761388.427697:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761388.427700:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761388.427703:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496030208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761388.427724:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496030336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761388.427741:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761388.427750:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761388/real 1789761388] req@ffff8dd205264000 x1876699496030336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761388.427758:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496030336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761388.427764:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761388.427767:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761388.427782:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496030336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761388.427789:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496030464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761388.427813:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761388.427825:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761388/real 1789761388] req@ffff8dd205265880 x1876699496030464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761388.427836:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496030464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761388.427846:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761388.427849:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761388.427851:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496030464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761388.427855:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496030592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761388.427870:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761388.427877:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761388/real 1789761388] req@ffff8dd205266a00 x1876699496030592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761388.427885:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496030592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761388.427890:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761388.427892:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761388.427894:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496030592:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761389.003344:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761390.027363:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761391.051340:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761392.075360:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761393.099333:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761393.547268:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761393.547278:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761393.547281:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761393.547290:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1373 00000100:00080000:2.0:1789761393.547300:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761393.547306:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761393.547345:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496030720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761393.547377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761393.547382:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761393.547384:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761393.547388:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1373 00000100:00080000:2.0:1789761393.547393:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761393.547397:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761393.547408:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761393.547425:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496030848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761393.547429:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496030720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761393.547447:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761393.547451:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761393.547454:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761393.547469:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1373 00000100:00080000:2.0:1789761393.547474:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761393.547479:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761393.547486:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761393.547491:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761393.547506:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496030976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761393.547518:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761393.547522:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761393.547524:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761393.547530:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761393/real 1789761393] req@ffff8dd205264700 x1876699496030720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761393.547538:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1373 00000100:00080000:2.0:1789761393.547545:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761393.547549:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496030720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761393.547554:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1789761393.547563:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789761393.547564:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761393.547569:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761393.547574:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496030720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761393.547583:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496031104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761393.547595:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496030848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761393.547624:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761393.547635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761393/real 1789761393] req@ffff8dd205266300 x1876699496030848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761393.547643:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496030848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761393.547649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761393.547651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761393.547653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496030848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761393.547657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496030976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761393.547668:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761393.547673:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761393/real 1789761393] req@ffff8dd205265180 x1876699496030976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761393.547680:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496030976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761393.547686:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761393.547687:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761393.547690:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496030976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761393.547698:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496031104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761393.547708:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761393.547714:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761393/real 1789761393] req@ffff8dd205266d80 x1876699496031104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761393.547721:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496031104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761393.547726:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761393.547728:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761393.547745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496031104:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761394.123393:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761395.147315:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761396.171308:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761397.195335:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761398.219358:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761398.667320:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761398.667349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761398.667353:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761398.667364:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1378 00000100:00080000:1.0:1789761398.667375:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761398.667382:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761398.667434:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496031232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761398.667461:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761398.667463:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761398.667465:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761398.667468:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1378 00000100:00080000:1.0:1789761398.667470:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761398.667473:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761398.667479:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761398.667489:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496031360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761398.667496:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761398.667499:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761398.667501:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761398.667503:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1378 00000100:00080000:1.0:1789761398.667506:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761398.667510:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761398.667513:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761398.667519:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496031488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761398.667528:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761398.667531:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761398.667532:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761398.667536:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1378 00000100:00080000:1.0:1789761398.667551:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761398.667557:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761398.667560:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496031232:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789761398.667562:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761398.667577:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496031616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761398.667660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761398.667729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761398/real 1789761398] req@ffff8dd20504c380 x1876699496031232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761398.667762:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496031232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761398.667772:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761398.667776:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761398.667779:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496031232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761398.667787:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496031360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761398.667814:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761398.667828:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761398/real 1789761398] req@ffff8dd20504d880 x1876699496031360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761398.667847:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496031360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761398.667853:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761398.667856:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761398.667858:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496031360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761398.667885:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496031488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761398.667903:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761398.667911:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761398/real 1789761398] req@ffff8dd20504ed80 x1876699496031488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761398.667919:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496031488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761398.667925:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761398.667928:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761398.667930:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496031488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761398.667934:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496031616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761398.667944:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761398.667951:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761398/real 1789761398] req@ffff8dd20504f100 x1876699496031616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761398.667958:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496031616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761398.667970:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761398.667973:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761398.667977:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496031616:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761399.243319:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761400.267337:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761401.291314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761402.315360:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761403.339297:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761403.787249:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761403.787259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761403.787263:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761403.787272:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1383 00000100:00080000:1.0:1789761403.787281:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761403.787288:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761403.787328:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496031744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761403.787349:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761403.787352:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761403.787353:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761403.787356:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1383 00000100:00080000:1.0:1789761403.787358:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761403.787361:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761403.787369:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761403.787378:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496031872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761403.787387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761403.787390:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496031744:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761403.787391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761403.787394:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761403.787401:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1383 00000100:00080000:1.0:1789761403.787403:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761403.787406:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761403.787409:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761403.787419:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496032000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761403.787426:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761403.787429:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761403.787431:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761403.787433:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1383 00000100:00080000:1.0:1789761403.787443:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761403.787446:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761403.787449:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761403.787453:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761403.787459:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496032128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761403.787491:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761403/real 1789761403] req@ffff8dd20504ce00 x1876699496031744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761403.787505:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496031744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761403.787512:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761403.787515:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761403.787522:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496031744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761403.787544:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496031872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761403.787568:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761403.787581:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761403/real 1789761403] req@ffff8dd20504f800 x1876699496031872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761403.787595:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496031872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761403.787604:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761403.787608:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761403.787612:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496031872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761403.787619:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496032000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761403.787640:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761403.787651:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761403/real 1789761403] req@ffff8dd20504fb80 x1876699496032000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761403.787677:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496032000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761403.787683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761403.787685:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761403.787687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496032000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761403.787691:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496032128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761403.787706:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761403.787731:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761403/real 1789761403] req@ffff8dd20504df80 x1876699496032128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761403.787743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496032128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761403.787753:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761403.787756:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761403.787760:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496032128:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761404.363348:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761405.387340:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761406.411365:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761407.435349:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761408.459363:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761408.907279:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761408.907288:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761408.907293:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761408.907302:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1388 00000100:00080000:2.0:1789761408.907311:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761408.907318:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761408.907348:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496032256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761408.907370:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761408.907373:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761408.907374:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761408.907379:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1388 00000100:00080000:2.0:1789761408.907382:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761408.907386:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761408.907396:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761408.907413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496032384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761408.907419:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496032256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761408.907436:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761408.907439:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761408.907441:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761408.907445:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1388 00000100:00080000:2.0:1789761408.907450:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761408.907455:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761408.907462:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761408.907476:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496032512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761408.907486:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761408.907496:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761408.907500:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761408.907502:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761408.907506:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1388 00000100:00080000:2.0:1789761408.907511:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761408.907514:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761408.907518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761408.907535:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496032640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761408.907546:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761408/real 1789761408] req@ffff8dd205265f80 x1876699496032256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761408.907568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496032256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761408.907579:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761408.907583:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761408.907587:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496032256:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761408.907606:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496032384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761408.907639:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761408.907651:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761408/real 1789761408] req@ffff8dd205266d80 x1876699496032384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761408.907660:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496032384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761408.907677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761408.907679:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761408.907681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496032384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761408.907686:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496032512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761408.907700:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761408.907707:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761408/real 1789761408] req@ffff8dd205265180 x1876699496032512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761408.907720:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496032512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761408.907725:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761408.907727:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761408.907729:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496032512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761408.907732:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496032640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761408.907743:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761408.907749:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761408/real 1789761408] req@ffff8dd205266300 x1876699496032640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761408.907756:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496032640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761408.907761:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761408.907763:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761408.907764:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496032640:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761409.483325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761410.507296:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761411.531380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761412.555334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761413.579353:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761414.027303:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761414.027316:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761414.027321:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761414.027334:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1393 00000100:00080000:0.0:1789761414.027345:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761414.027354:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761414.027397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496032768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761414.027422:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761414.027425:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761414.027426:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761414.027430:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1393 00000100:00080000:0.0:1789761414.027433:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761414.027436:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761414.027443:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761414.027453:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496032896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761414.027460:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761414.027463:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761414.027464:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761414.027467:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1393 00000100:00080000:0.0:1789761414.027470:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761414.027475:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761414.027475:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496032768:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789761414.027482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761414.027491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496033024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761414.027499:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761414.027501:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761414.027502:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761414.027506:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1393 00000100:00080000:0.0:1789761414.027508:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761414.027512:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761414.027516:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761414.027524:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496033152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761414.027565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496032896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761414.027614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496033024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761414.027632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496033152:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761414.603327:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761415.627358:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761416.651318:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761417.099445:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789761417.099490:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761417.099503:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761417.099536:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761417.099542:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761417.099549:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789761417.099597:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761414/real 1789761417] req@ffff8dd211836d80 x1876699496033024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761417.099625:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496033024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761417.099638:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761417.099643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761417.099648:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496033024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761417.099656:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761414/real 1789761417] req@ffff8dd211834e00 x1876699496033152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761417.099673:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496033152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761417.099680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761417.099683:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761417.099687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496033152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761417.099694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761414/real 1789761417] req@ffff8dd211837480 x1876699496032768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761417.099713:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496032768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761417.099721:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761417.099725:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761417.099729:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496032768:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761417.099736:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761414/real 1789761417] req@ffff8dd211835180 x1876699496032896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761417.099747:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496032896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761417.099755:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761417.099758:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761417.099762:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496032896:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761417.675369:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761418.699314:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761419.147225:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761419.147234:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761419.147238:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761419.147247:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1398 00000100:00080000:0.0:1789761419.147257:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761419.147263:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761419.147294:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496033280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761419.147318:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761419.147321:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761419.147323:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761419.147325:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1398 00000100:00080000:0.0:1789761419.147328:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761419.147331:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761419.147339:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761419.147348:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496033408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761419.147357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761419.147359:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761419.147361:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761419.147365:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1398 00000100:00080000:0.0:1789761419.147368:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761419.147368:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496033280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761419.147372:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761419.147379:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761419.147386:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496033536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761419.147393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761419.147397:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761419.147399:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761419.147403:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1398 00000100:00080000:0.0:1789761419.147407:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761419.147411:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761419.147417:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761419.147428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496033664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761419.147441:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761419.147484:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761419/real 1789761419] req@ffff8dd211835180 x1876699496033280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761419.147504:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496033280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761419.147515:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761419.147520:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761419.147524:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496033280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761419.147565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496033408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761419.147592:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761419.147608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761419/real 1789761419] req@ffff8dd211837480 x1876699496033408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761419.147622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496033408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761419.147630:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761419.147634:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761419.147637:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496033408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761419.147644:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496033536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761419.147666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761419.147678:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761419/real 1789761419] req@ffff8dd211834e00 x1876699496033536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761419.147691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496033536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761419.147699:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761419.147702:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761419.147706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496033536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761419.147712:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496033664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761419.147733:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761419.147745:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761419/real 1789761419] req@ffff8dd211836d80 x1876699496033664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761419.147757:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496033664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761419.147766:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761419.147769:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761419.147772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496033664:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761419.723363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761420.747302:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761421.771328:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761422.795298:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761423.819314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761424.267255:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761424.267265:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761424.267270:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761424.267280:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1403 00000100:00080000:3.0:1789761424.267294:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761424.267301:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761424.267353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496033792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761424.267386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761424.267391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761424.267393:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761424.267400:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1403 00000100:00080000:3.0:1789761424.267407:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761424.267413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761424.267426:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761424.267431:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496033792:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761424.267449:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496033920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761424.267465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761424.267469:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761424.267472:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761424.267477:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1403 00000100:00080000:3.0:1789761424.267483:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761424.267489:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761424.267496:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789761424.267516:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761424.267533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496034048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761424.267551:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761424.267555:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761424.267558:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761424.267563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761424/real 1789761424] req@ffff8dd203145f80 x1876699496033792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761424.267567:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1403 00000100:00080000:3.0:1789761424.267576:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761424.267587:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761424.267596:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761424.267597:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496033792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761424.267619:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761424.267620:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496034176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761424.267627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761424.267634:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496033792:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761424.267671:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496033920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761424.267695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761424.267706:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761424/real 1789761424] req@ffff8dd203145180 x1876699496033920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761424.267715:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496033920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761424.267721:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761424.267724:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761424.267726:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496033920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761424.267731:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496034048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761424.267751:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761424.267762:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761424/real 1789761424] req@ffff8dd203145c00 x1876699496034048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761424.267784:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496034048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761424.267791:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761424.267794:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761424.267796:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496034048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761424.267800:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496034176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761424.267816:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761424.267823:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761424/real 1789761424] req@ffff8dd203146300 x1876699496034176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761424.267831:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496034176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761424.267836:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761424.267838:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761424.267840:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496034176:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761424.843299:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761425.867336:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761426.891356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761427.915359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761428.939390:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761429.387309:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761429.387319:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761429.387323:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761429.387332:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1408 00000100:00080000:3.0:1789761429.387342:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761429.387348:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761429.387381:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496034304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761429.387406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761429.387409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761429.387411:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761429.387415:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1408 00000100:00080000:3.0:1789761429.387419:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761429.387424:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761429.387433:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761429.387450:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496034304:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761429.387451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496034432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761429.387480:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761429.387484:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761429.387486:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761429.387490:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1408 00000100:00080000:3.0:1789761429.387494:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761429.387498:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761429.387503:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761429.387514:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496034560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761429.387522:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761429.387525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761429.387526:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761429.387530:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761429.387533:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1408 00000100:00080000:3.0:1789761429.387538:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761429.387541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761429.387545:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761429.387554:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496034688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761429.387575:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761429/real 1789761429] req@ffff8dd203147480 x1876699496034304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761429.387591:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496034304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761429.387604:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761429.387607:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761429.387610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496034304:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761429.387631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496034432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761429.387646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761429.387655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761429/real 1789761429] req@ffff8dd203144000 x1876699496034432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761429.387663:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496034432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761429.387668:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761429.387671:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761429.387673:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496034432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761429.387676:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496034560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761429.387687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761429.387693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761429/real 1789761429] req@ffff8dd203147100 x1876699496034560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761429.387700:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496034560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761429.387705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761429.387707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761429.387709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496034560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761429.387713:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496034688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761429.387723:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761429.387729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761429/real 1789761429] req@ffff8dd203147800 x1876699496034688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761429.387737:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496034688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761429.387742:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761429.387743:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761429.387745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496034688:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761429.963285:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761430.987346:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761432.011330:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761433.035343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761434.059319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761434.507251:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761434.507260:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761434.507264:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761434.507273:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1414 00000100:00080000:3.0:1789761434.507282:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761434.507288:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761434.507328:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496034816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761434.507349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761434.507352:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761434.507353:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761434.507356:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1414 00000100:00080000:3.0:1789761434.507359:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761434.507361:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761434.507369:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761434.507379:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496034944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761434.507386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761434.507388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761434.507390:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761434.507393:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1414 00000100:00080000:3.0:1789761434.507395:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761434.507398:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761434.507400:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761434.507407:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496035072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761434.507415:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496034816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761434.507417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761434.507437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761434.507439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761434.507443:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1414 00000100:00080000:3.0:1789761434.507446:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761434.507448:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761434.507453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761434.507462:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496035200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761434.507500:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761434.507543:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761434/real 1789761434] req@ffff8dd203145880 x1876699496034816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761434.507563:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496034816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761434.507575:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761434.507579:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761434.507584:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496034816:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761434.507591:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496034944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761434.507615:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761434.507625:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761434/real 1789761434] req@ffff8dd203147800 x1876699496034944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761434.507634:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496034944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761434.507641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761434.507644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761434.507648:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496034944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761434.507673:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496035072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761434.507696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761434.507705:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761434/real 1789761434] req@ffff8dd203147100 x1876699496035072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761434.507713:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496035072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761434.507721:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761434.507725:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761434.507728:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496035072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761434.507734:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496035200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761434.507754:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761434.507766:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761434/real 1789761434] req@ffff8dd203144000 x1876699496035200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761434.507779:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496035200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761434.507790:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761434.507792:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761434.507794:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496035200:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761435.083338:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761436.107314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761437.131354:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761438.155291:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761439.179375:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761439.627279:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761439.627288:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761439.627293:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761439.627304:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1419 00000100:00080000:3.0:1789761439.627317:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761439.627325:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761439.627376:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496035328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761439.627419:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761439.627422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761439.627424:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761439.627428:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1419 00000100:00080000:3.0:1789761439.627433:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761439.627438:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761439.627448:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761439.627459:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496035328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761439.627465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496035456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761439.627481:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761439.627484:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761439.627485:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761439.627489:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1419 00000100:00080000:3.0:1789761439.627492:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761439.627495:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761439.627500:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761439.627510:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496035584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761439.627519:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761439.627522:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761439.627523:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761439.627526:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1419 00000100:00080000:3.0:1789761439.627529:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761439.627536:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761439.627541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761439.627548:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761439.627560:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496035712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761439.627587:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761439/real 1789761439] req@ffff8dd203147480 x1876699496035328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761439.627603:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496035328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761439.627619:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761439.627622:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761439.627625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496035328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761439.627647:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496035456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761439.627666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761439.627675:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761439/real 1789761439] req@ffff8dd203146300 x1876699496035456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761439.627683:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496035456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761439.627689:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761439.627692:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761439.627694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496035456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761439.627699:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496035584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761439.627711:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761439.627717:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761439/real 1789761439] req@ffff8dd203145c00 x1876699496035584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761439.627725:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496035584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761439.627731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761439.627732:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761439.627734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496035584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761439.627739:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496035712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761439.627750:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761439.627756:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761439/real 1789761439] req@ffff8dd203145180 x1876699496035712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761439.627763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496035712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761439.627768:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761439.627771:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761439.627772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496035712:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761440.203380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761441.227323:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761442.251296:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761443.275337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761444.299350:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761444.747284:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761444.747294:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761444.747298:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761444.747307:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1424 00000100:00080000:3.0:1789761444.747316:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761444.747323:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761444.747371:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496035840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761444.747398:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761444.747401:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761444.747402:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761444.747405:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1424 00000100:00080000:3.0:1789761444.747409:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761444.747412:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761444.747419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761444.747430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496035968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761444.747438:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761444.747441:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761444.747442:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761444.747447:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1424 00000100:00100000:1.0:1789761444.747449:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496035840:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761444.747450:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761444.747456:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761444.747461:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761444.747469:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496036096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761444.747475:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761444.747478:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761444.747479:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761444.747482:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1424 00000100:00080000:3.0:1789761444.747484:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761444.747487:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761444.747490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761444.747496:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496036224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761444.747509:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761444.747545:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761444/real 1789761444] req@ffff8dd203145f80 x1876699496035840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761444.747559:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496035840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761444.747567:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761444.747570:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761444.747573:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496035840:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761444.747598:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496035968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761444.747615:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761444.747624:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761444/real 1789761444] req@ffff8dd203144380 x1876699496035968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761444.747632:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496035968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761444.747638:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761444.747640:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761444.747642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496035968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761444.747646:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496036096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761444.747657:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761444.747663:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761444/real 1789761444] req@ffff8dd203144a80 x1876699496036096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761444.747671:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496036096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761444.747680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761444.747682:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761444.747685:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496036096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761444.747688:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496036224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761444.747697:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761444.747703:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761444/real 1789761444] req@ffff8dd203144700 x1876699496036224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761444.747710:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496036224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761444.747715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761444.747717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761444.747719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496036224:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761445.323351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761446.347365:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761447.371346:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761448.395365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761449.419366:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761449.867275:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761449.867284:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761449.867289:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761449.867300:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1429 00000100:00080000:3.0:1789761449.867326:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761449.867335:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761449.867382:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496036352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761449.867411:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761449.867415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761449.867416:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761449.867420:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1429 00000100:00080000:3.0:1789761449.867423:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761449.867426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761449.867435:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761449.867446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496036480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761449.867455:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761449.867462:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496036352:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761449.867464:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761449.867468:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761449.867473:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1429 00000100:00080000:3.0:1789761449.867476:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761449.867479:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761449.867483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761449.867492:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496036608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761449.867499:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761449.867502:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761449.867503:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761449.867506:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1429 00000100:00080000:3.0:1789761449.867508:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761449.867511:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761449.867514:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761449.867521:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496036736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761449.867551:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761449.867601:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761449/real 1789761449] req@ffff8dd203145500 x1876699496036352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761449.867619:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496036352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761449.867628:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761449.867632:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761449.867635:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496036352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761449.867658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496036480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761449.867694:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761449.867710:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761449/real 1789761449] req@ffff8dd203146a00 x1876699496036480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761449.867725:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496036480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761449.867735:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761449.867738:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761449.867741:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496036480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761449.867746:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496036608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761449.867770:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761449.867778:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761449/real 1789761449] req@ffff8dd203144e00 x1876699496036608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761449.867786:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496036608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761449.867792:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761449.867795:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761449.867797:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496036608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761449.867801:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496036736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761449.867812:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761449.867818:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761449/real 1789761449] req@ffff8dd203146d80 x1876699496036736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761449.867826:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496036736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761449.867832:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761449.867835:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761449.867838:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496036736:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761450.443350:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761451.467348:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761452.491328:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761453.515315:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761454.539361:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761454.987295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761454.987304:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761454.987308:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761454.987317:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1434 00000100:00080000:3.0:1789761454.987327:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761454.987334:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761454.987367:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496036864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761454.987386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761454.987389:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761454.987390:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761454.987393:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1434 00000100:00080000:3.0:1789761454.987396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761454.987399:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761454.987411:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761454.987420:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496036864:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761454.987424:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496036992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761454.987433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761454.987436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761454.987438:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761454.987440:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1434 00000100:00080000:3.0:1789761454.987443:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761454.987445:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761454.987449:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761454.987457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496037120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761454.987464:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761454.987468:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761454.987469:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761454.987471:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1434 00000100:00080000:3.0:1789761454.987473:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761454.987476:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761454.987480:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1789761454.987482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761454.987491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496037248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761454.987517:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761454/real 1789761454] req@ffff8dd203147b80 x1876699496036864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761454.987533:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496036864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761454.987541:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761454.987544:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761454.987547:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496036864:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761454.987565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496036992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761454.987579:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761454.987587:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761454/real 1789761454] req@ffff8dd203146d80 x1876699496036992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761454.987595:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496036992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761454.987605:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761454.987608:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761454.987610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496036992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761454.987613:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496037120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761454.987624:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761454.987630:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761454/real 1789761454] req@ffff8dd203144e00 x1876699496037120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761454.987637:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496037120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761454.987642:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761454.987644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761454.987646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496037120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761454.987649:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496037248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761454.987659:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761454.987673:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761454/real 1789761454] req@ffff8dd203146a00 x1876699496037248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761454.987680:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496037248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761454.987685:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761454.987687:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761454.987690:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496037248:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761455.563331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761456.587367:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761457.611333:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761458.635379:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761459.659304:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761460.107257:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761460.107267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761460.107270:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761460.107279:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1439 00000100:00080000:3.0:1789761460.107289:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761460.107294:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761460.107335:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496037376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761460.107366:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761460.107369:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761460.107370:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761460.107374:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1439 00000100:00080000:3.0:1789761460.107377:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761460.107381:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761460.107389:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761460.107401:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496037376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761460.107403:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496037504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761460.107415:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761460.107417:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761460.107418:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761460.107421:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1439 00000100:00080000:3.0:1789761460.107423:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761460.107426:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761460.107431:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761460.107439:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496037632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761460.107446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761460.107449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761460.107451:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761460.107453:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1439 00000100:00080000:3.0:1789761460.107455:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761460.107457:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761460.107460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761460.107461:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761460.107474:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496037760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761460.107501:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761460/real 1789761460] req@ffff8dd203145500 x1876699496037376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761460.107516:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496037376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761460.107523:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761460.107526:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761460.107529:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496037376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761460.107548:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496037504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761460.107563:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761460.107571:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761460/real 1789761460] req@ffff8dd203144700 x1876699496037504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761460.107579:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496037504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761460.107585:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761460.107587:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761460.107590:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496037504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761460.107593:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496037632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761460.107604:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761460.107611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761460/real 1789761460] req@ffff8dd203144a80 x1876699496037632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761460.107618:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496037632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761460.107623:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761460.107625:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761460.107627:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496037632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761460.107631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496037760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761460.107641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761460.107646:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761460/real 1789761460] req@ffff8dd203144380 x1876699496037760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761460.107654:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496037760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761460.107659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761460.107660:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761460.107662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496037760:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761460.683381:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761461.707322:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761462.731362:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761463.755315:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761464.779352:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761465.227314:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761465.227330:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761465.227334:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761465.227343:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1444 00000100:00080000:0.0:1789761465.227352:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761465.227361:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761465.227395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496037888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761465.227421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761465.227423:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761465.227425:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761465.227428:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1444 00000100:00080000:0.0:1789761465.227430:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761465.227432:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761465.227440:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761465.227450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496038016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761465.227459:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761465.227463:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789761465.227464:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496037888:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761465.227465:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761465.227472:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1444 00000100:00080000:0.0:1789761465.227475:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761465.227477:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761465.227480:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761465.227494:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496038144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761465.227502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761465.227505:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761465.227506:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761465.227509:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1444 00000100:00080000:0.0:1789761465.227523:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761465.227526:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761465.227529:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761465.227533:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761465.227541:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496038272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761465.227574:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761465/real 1789761465] req@ffff8dd211836d80 x1876699496037888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761465.227587:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496037888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761465.227595:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761465.227598:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761465.227601:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496037888:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761465.227619:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496038016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761465.227641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761465.227654:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761465/real 1789761465] req@ffff8dd211834e00 x1876699496038016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761465.227668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496038016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761465.227677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761465.227681:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761465.227685:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496038016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761465.227691:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496038144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761465.227711:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761465.227719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761465/real 1789761465] req@ffff8dd211837480 x1876699496038144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761465.227727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496038144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761465.227732:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761465.227735:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761465.227737:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496038144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761465.227741:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496038272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761465.227752:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761465.227758:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761465/real 1789761465] req@ffff8dd211835180 x1876699496038272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761465.227765:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496038272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761465.227770:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761465.227772:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761465.227774:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496038272:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761465.803351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761466.827391:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761467.851356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761468.875363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761469.899334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761470.347417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761470.347430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761470.347435:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761470.347446:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1449 00000100:00080000:3.0:1789761470.347458:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761470.347482:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761470.347524:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496038400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761470.347562:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761470.347567:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761470.347569:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761470.347574:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1449 00000100:00080000:3.0:1789761470.347596:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761470.347601:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761470.347613:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761470.347632:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496038528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761470.347639:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496038400:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761470.347657:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761470.347662:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761470.347665:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761470.347686:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1449 00000100:00080000:3.0:1789761470.347690:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761470.347694:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761470.347701:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761470.347714:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496038656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761470.347724:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761470.347733:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761470.347738:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761470.347740:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761470.347746:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1449 00000100:00080000:3.0:1789761470.347750:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761470.347755:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761470.347761:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761470.347768:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761470/real 1789761470] req@ffff8dd203145f80 x1876699496038400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761470.347777:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496038784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761470.347788:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496038400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761470.347835:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761470.347840:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761470.347844:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496038400:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761470.347868:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496038528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761470.347888:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761470.347898:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761470/real 1789761470] req@ffff8dd203145180 x1876699496038528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761470.347908:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496038528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761470.347925:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761470.347928:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761470.347930:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496038528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761470.347951:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496038656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761470.347965:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761470.347973:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761470/real 1789761470] req@ffff8dd203145c00 x1876699496038656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761470.347982:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496038656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761470.347987:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761470.347989:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761470.347992:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496038656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761470.347995:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496038784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761470.348023:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761470.348029:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761470/real 1789761470] req@ffff8dd203146300 x1876699496038784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761470.348038:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496038784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761470.348049:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761470.348051:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761470.348054:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496038784:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761470.923339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761471.947362:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761472.971343:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761473.995318:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761475.019320:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761475.467285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761475.467296:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761475.467300:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761475.467312:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1455 00000100:00080000:1.0:1789761475.467331:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761475.467339:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761475.467377:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496038912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761475.467406:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761475.467410:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761475.467413:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761475.467417:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1455 00000100:00080000:1.0:1789761475.467422:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761475.467427:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761475.467437:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761475.467449:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496038912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761475.467453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496039040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761475.467468:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761475.467472:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761475.467475:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761475.467479:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1455 00000100:00080000:1.0:1789761475.467483:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761475.467488:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761475.467495:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761475.467507:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496039168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761475.467516:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761475.467523:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761475.467527:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761475.467530:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761475.467535:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1455 00000100:00080000:1.0:1789761475.467539:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761475.467544:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761475.467550:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761475.467557:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761475/real 1789761475] req@ffff8dd20504d500 x1876699496038912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761475.467564:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496039296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761475.467577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496038912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761475.467595:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761475.467600:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761475.467605:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496038912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761475.467631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496039040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761475.467673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761475.467684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761475/real 1789761475] req@ffff8dd20504e300 x1876699496039040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761475.467704:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496039040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761475.467730:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761475.467733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761475.467737:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496039040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761475.467744:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496039168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761475.467772:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761475.467782:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761475/real 1789761475] req@ffff8dd20504c700 x1876699496039168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761475.467790:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496039168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761475.467795:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761475.467797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761475.467800:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496039168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761475.467804:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496039296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761475.467815:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761475.467821:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761475/real 1789761475] req@ffff8dd20504e680 x1876699496039296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761475.467829:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496039296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761475.467838:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761475.467840:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761475.467842:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496039296:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761476.043347:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761477.067333:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761478.091371:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761479.115288:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761480.139330:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761480.587289:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761480.587298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761480.587302:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761480.587311:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1460 00000100:00080000:3.0:1789761480.587321:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761480.587327:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761480.587363:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496039424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761480.587395:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761480.587399:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761480.587401:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761480.587406:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1460 00000100:00080000:3.0:1789761480.587410:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761480.587413:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761480.587421:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761480.587432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496039552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761480.587439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761480.587442:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761480.587443:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761480.587446:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1460 00000100:00080000:3.0:1789761480.587448:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761480.587450:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761480.587454:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761480.587460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496039680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761480.587467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761480.587469:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761480.587471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761480.587475:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1460 00000100:00080000:3.0:1789761480.587478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761480.587482:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761480.587487:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761480.587501:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496039424:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761480.587502:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496039808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761480.587590:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496039552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761480.587610:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496039680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761480.587635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496039808:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761481.163265:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761482.187288:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761483.211298:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761483.659420:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789761483.659462:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761483.659482:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761483.659516:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761483.659526:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761483.659534:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789761483.659576:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761480/real 1789761483] req@ffff8dd203147800 x1876699496039808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761483.659596:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496039808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761483.659605:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761483.659609:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761483.659612:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496039808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761483.659617:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761480/real 1789761483] req@ffff8dd203147480 x1876699496039424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761483.659625:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496039424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761483.659631:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761483.659634:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761483.659638:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496039424:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761483.659650:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761480/real 1789761483] req@ffff8dd203144000 x1876699496039552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761483.659662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496039552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761483.659670:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761483.659673:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761483.659677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496039552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761483.659683:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761480/real 1789761483] req@ffff8dd203147100 x1876699496039680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761483.659696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496039680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761483.659704:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761483.659708:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761483.659711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496039680:192.168.203.109@tcp:8:kworker.0 00010000:00010000:2.0:1789761484.235293:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761485.259377:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761485.707242:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761485.707254:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761485.707258:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761485.707269:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1465 00000100:00080000:3.0:1789761485.707280:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761485.707288:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761485.707332:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496039936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761485.707360:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761485.707367:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761485.707369:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761485.707372:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1465 00000100:00080000:3.0:1789761485.707376:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761485.707380:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761485.707388:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761485.707401:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496040064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761485.707411:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761485.707412:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496039936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761485.707416:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761485.707422:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761485.707427:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1465 00000100:00080000:3.0:1789761485.707431:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761485.707434:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761485.707440:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761485.707454:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496040192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761485.707464:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761485.707478:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761485.707480:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761485.707484:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1465 00000800:00000100:0.0:1789761485.707490:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761485.707492:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761485.707500:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761485.707506:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761485.707526:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496040320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761485.707530:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761485/real 1789761485] req@ffff8dd203145880 x1876699496039936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761485.707570:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496039936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761485.707582:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761485.707592:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761485.707596:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496039936:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761485.707622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496040064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761485.707650:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761485.707664:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761485/real 1789761485] req@ffff8dd203147100 x1876699496040064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761485.707677:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496040064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761485.707686:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761485.707689:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761485.707692:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496040064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761485.707698:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496040192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761485.707720:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761485.707733:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761485/real 1789761485] req@ffff8dd203144000 x1876699496040192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761485.707746:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496040192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761485.707762:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761485.707765:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761485.707769:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496040192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761485.707776:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496040320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761485.707797:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761485.707810:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761485/real 1789761485] req@ffff8dd203147480 x1876699496040320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761485.707822:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496040320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761485.707830:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761485.707834:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761485.707838:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496040320:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761486.283290:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761487.307295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761488.331320:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761489.355384:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761490.379310:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761490.827277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761490.827293:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761490.827298:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761490.827309:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1470 00000100:00080000:0.0:1789761490.827321:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761490.827329:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761490.827365:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496040448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761490.827388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761490.827391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761490.827393:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761490.827397:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1470 00000100:00080000:0.0:1789761490.827401:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761490.827405:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761490.827415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761490.827430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496040576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761490.827443:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761490.827446:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496040448:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761490.827448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761490.827454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761490.827459:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1470 00000100:00080000:0.0:1789761490.827464:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761490.827468:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761490.827475:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761490.827497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496040704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761490.827506:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761490.827515:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761490.827520:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761490.827522:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761490.827527:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1470 00000100:00080000:0.0:1789761490.827531:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761490.827536:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761490.827542:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761490.827543:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761490/real 1789761490] req@ffff8dd211834a80 x1876699496040448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761490.827559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496040832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761490.827561:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496040448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761490.827593:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761490.827598:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761490.827601:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496040448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761490.827622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496040576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761490.827638:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761490.827647:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761490/real 1789761490] req@ffff8dd211836680 x1876699496040576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761490.827655:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496040576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761490.827661:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761490.827664:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761490.827666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496040576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761490.827669:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496040704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761490.827681:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761490.827687:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761490/real 1789761490] req@ffff8dd211834380 x1876699496040704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761490.827694:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496040704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761490.827704:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761490.827707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761490.827709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496040704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761490.827712:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496040832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761490.827729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761490.827751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761490/real 1789761490] req@ffff8dd211835f80 x1876699496040832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761490.827765:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496040832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761490.827774:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761490.827777:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761490.827781:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496040832:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761491.403343:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761492.427318:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761493.451350:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761494.475305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761495.499308:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761495.947303:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761495.947317:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761495.947322:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761495.947332:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1475 00000100:00080000:3.0:1789761495.947344:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761495.947350:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761495.947384:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496040960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761495.947408:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761495.947412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761495.947413:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761495.947416:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1475 00000100:00080000:3.0:1789761495.947419:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761495.947421:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761495.947430:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761495.947443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496041088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761495.947457:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761495.947460:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496040960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761495.947461:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761495.947466:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761495.947473:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1475 00000100:00080000:3.0:1789761495.947478:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761495.947482:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761495.947490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761495.947506:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496041216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761495.947518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761495.947519:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761495.947525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761495.947528:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761495.947533:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1475 00000100:00080000:3.0:1789761495.947536:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761495.947541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761495.947546:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761495.947556:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496041344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761495.947568:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761495/real 1789761495] req@ffff8dd203147800 x1876699496040960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761495.947583:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496040960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761495.947592:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761495.947595:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761495.947598:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496040960:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761495.947616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496041088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761495.947633:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761495.947641:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761495/real 1789761495] req@ffff8dd203146300 x1876699496041088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761495.947650:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496041088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761495.947656:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761495.947658:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761495.947660:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496041088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761495.947664:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496041216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761495.947675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761495.947681:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761495/real 1789761495] req@ffff8dd203145c00 x1876699496041216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761495.947688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496041216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761495.947694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761495.947696:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761495.947697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496041216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761495.947701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496041344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761495.947711:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761495.947716:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761495/real 1789761495] req@ffff8dd203145180 x1876699496041344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761495.947724:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496041344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761495.947729:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761495.947731:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761495.947733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496041344:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761496.523313:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761497.547314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761498.571344:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761499.595330:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761500.619321:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761501.067283:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761501.067295:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761501.067299:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761501.067308:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1480 00000100:00080000:3.0:1789761501.067317:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761501.067323:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761501.067357:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496041472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761501.067381:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761501.067384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761501.067385:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761501.067388:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1480 00000100:00080000:3.0:1789761501.067390:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761501.067393:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761501.067400:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761501.067410:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496041600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761501.067422:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761501.067425:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761501.067428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761501.067432:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1480 00000100:00100000:1.0:1789761501.067439:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496041472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761501.067439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761501.067446:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761501.067454:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761501.067468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496041728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761501.067477:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761501.067480:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761501.067481:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761501.067485:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1480 00000100:00080000:3.0:1789761501.067490:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761501.067494:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761501.067499:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761501.067511:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761501.067514:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496041856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761501.067552:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761501/real 1789761501] req@ffff8dd203145f80 x1876699496041472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761501.067568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496041472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761501.067576:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761501.067579:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761501.067582:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496041472:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761501.067588:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496041600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761501.067610:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761501.067618:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761501/real 1789761501] req@ffff8dd203144380 x1876699496041600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761501.067627:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496041600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761501.067637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761501.067640:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761501.067643:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496041600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761501.067662:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496041728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761501.067674:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761501.067680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761501/real 1789761501] req@ffff8dd203144a80 x1876699496041728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761501.067688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496041728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761501.067693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761501.067695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761501.067697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496041728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761501.067700:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496041856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761501.067710:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761501.067716:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761501/real 1789761501] req@ffff8dd203144700 x1876699496041856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761501.067723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496041856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761501.067728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761501.067730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761501.067732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496041856:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761501.643359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761502.667330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761503.691363:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761504.715299:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761505.739322:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761506.187314:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761506.187324:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761506.187327:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761506.187336:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1485 00000100:00080000:3.0:1789761506.187346:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761506.187352:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761506.187388:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496041984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761506.187413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761506.187416:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761506.187417:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761506.187420:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1485 00000100:00080000:3.0:1789761506.187422:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761506.187424:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761506.187432:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761506.187442:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496042112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761506.187449:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761506.187452:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761506.187453:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761506.187457:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1485 00000100:00080000:3.0:1789761506.187459:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761506.187462:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761506.187468:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761506.187477:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496041984:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761506.187477:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496042240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761506.187501:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761506.187504:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761506.187505:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761506.187507:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1485 00000100:00080000:3.0:1789761506.187508:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761506.187511:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761506.187514:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761506.187529:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496042368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761506.187583:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761506.187627:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761506/real 1789761506] req@ffff8dd203145500 x1876699496041984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761506.187651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496041984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761506.187662:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761506.187666:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761506.187669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496041984:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761506.187675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496042112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761506.187694:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761506.187704:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761506/real 1789761506] req@ffff8dd203146a00 x1876699496042112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761506.187713:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496042112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761506.187719:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761506.187722:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761506.187725:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496042112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761506.187753:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496042240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761506.187766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761506.187773:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761506/real 1789761506] req@ffff8dd203144e00 x1876699496042240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761506.187781:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496042240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761506.187787:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761506.187790:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761506.187792:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496042240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761506.187796:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496042368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761506.187808:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761506.187815:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761506/real 1789761506] req@ffff8dd203146d80 x1876699496042368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761506.187824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496042368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761506.187830:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761506.187832:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761506.187834:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496042368:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761506.763367:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761507.787376:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761508.811331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761509.835318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761510.859344:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761511.307255:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761511.307263:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761511.307266:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761511.307275:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1490 00000100:00080000:3.0:1789761511.307285:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761511.307291:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761511.307324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496042496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761511.307361:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761511.307364:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761511.307365:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761511.307368:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1490 00000100:00080000:3.0:1789761511.307371:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761511.307374:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761511.307382:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761511.307392:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496042624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761511.307402:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496042496:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761511.307402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761511.307410:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761511.307413:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761511.307416:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1490 00000100:00080000:3.0:1789761511.307418:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761511.307421:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761511.307424:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761511.307432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496042752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761511.307439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761511.307442:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761511.307444:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761511.307447:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1490 00000100:00080000:3.0:1789761511.307450:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761511.307455:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761511.307460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761511.307465:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761511.307475:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496042880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761511.307503:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761511/real 1789761511] req@ffff8dd203147b80 x1876699496042496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761511.307524:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496042496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761511.307538:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761511.307542:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761511.307547:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496042496:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761511.307568:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496042624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761511.307595:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761511.307604:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761511/real 1789761511] req@ffff8dd203146d80 x1876699496042624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761511.307613:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496042624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761511.307620:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761511.307622:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761511.307624:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496042624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761511.307628:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496042752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761511.307650:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761511.307657:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761511/real 1789761511] req@ffff8dd203144e00 x1876699496042752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761511.307665:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496042752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761511.307670:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761511.307672:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761511.307674:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496042752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761511.307678:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496042880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761511.307688:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761511.307694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761511/real 1789761511] req@ffff8dd203146a00 x1876699496042880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761511.307702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496042880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761511.307707:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761511.307709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761511.307711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496042880:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761511.883315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761512.907290:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761513.931326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761514.955359:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761515.979305:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761516.427249:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761516.427263:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761516.427268:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761516.427280:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1496 00000100:00080000:3.0:1789761516.427291:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761516.427298:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761516.427331:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496043008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761516.427353:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761516.427356:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761516.427357:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761516.427360:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1496 00000100:00080000:3.0:1789761516.427363:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761516.427365:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761516.427373:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761516.427387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496043136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761516.427400:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496043008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761516.427401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761516.427409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761516.427412:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761516.427417:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1496 00000100:00080000:3.0:1789761516.427422:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761516.427427:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761516.427434:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761516.427448:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496043264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761516.427459:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761516.427463:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761516.427466:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761516.427467:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761516.427472:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1496 00000100:00080000:3.0:1789761516.427477:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761516.427481:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761516.427486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761516.427495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496043392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761516.427517:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761516/real 1789761516] req@ffff8dd203145500 x1876699496043008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761516.427532:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496043008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761516.427540:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761516.427543:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761516.427546:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496043008:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761516.427565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496043136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761516.427583:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761516.427592:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761516/real 1789761516] req@ffff8dd203144700 x1876699496043136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761516.427600:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496043136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761516.427606:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761516.427608:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761516.427610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496043136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761516.427614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496043264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761516.427625:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761516.427631:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761516/real 1789761516] req@ffff8dd203144a80 x1876699496043264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761516.427638:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496043264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761516.427643:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761516.427645:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761516.427647:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496043264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761516.427650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496043392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761516.427660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761516.427666:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761516/real 1789761516] req@ffff8dd203144380 x1876699496043392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761516.427673:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496043392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761516.427683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761516.427685:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761516.427686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496043392:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761517.003351:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761518.027314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761519.051332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761520.075284:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761521.099367:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761521.547242:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761521.547253:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761521.547264:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761521.547276:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1501 00000100:00080000:3.0:1789761521.547287:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761521.547295:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761521.547349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496043520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761521.547374:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761521.547377:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761521.547379:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761521.547382:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1501 00000100:00080000:3.0:1789761521.547386:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761521.547389:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761521.547397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761521.547411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496043648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761521.547421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761521.547424:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496043520:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761521.547424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761521.547426:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761521.547434:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1501 00000100:00080000:3.0:1789761521.547438:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761521.547441:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761521.547445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761521.547461:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496043776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761521.547474:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761521.547478:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761521.547480:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761521.547485:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1501 00000100:00080000:3.0:1789761521.547489:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761521.547494:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761521.547500:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1789761521.547503:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761521.547517:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496043904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761521.547544:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761521/real 1789761521] req@ffff8dd203145f80 x1876699496043520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761521.547559:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496043520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761521.547568:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761521.547571:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761521.547574:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496043520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761521.547595:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496043648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761521.547611:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761521.547620:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761521/real 1789761521] req@ffff8dd203145180 x1876699496043648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761521.547629:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496043648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761521.547635:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761521.547637:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761521.547639:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496043648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761521.547644:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496043776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761521.547656:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761521.547663:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761521/real 1789761521] req@ffff8dd203145c00 x1876699496043776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761521.547671:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496043776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761521.547677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761521.547679:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761521.547681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496043776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761521.547685:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496043904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761521.547695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761521.547702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761521/real 1789761521] req@ffff8dd203146300 x1876699496043904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761521.547709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496043904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761521.547715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761521.547717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761521.547719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496043904:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761522.123309:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761523.147332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761524.171351:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761525.195372:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761526.219337:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761526.667304:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761526.667323:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761526.667327:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761526.667335:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1506 00000100:00080000:0.0:1789761526.667346:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761526.667351:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761526.667384:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496044032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761526.667407:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761526.667411:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761526.667413:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761526.667417:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1506 00000100:00080000:0.0:1789761526.667422:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761526.667426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761526.667436:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761526.667450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496044160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761526.667454:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496044032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761526.667461:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761526.667464:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761526.667466:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761526.667470:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1506 00000100:00080000:0.0:1789761526.667474:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761526.667478:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761526.667483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761526.667496:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496044288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761526.667506:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761526.667509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761526.667511:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761526.667515:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1506 00000100:00080000:0.0:1789761526.667518:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761526.667523:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761526.667526:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789761526.667528:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761526.667537:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496044416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761526.667567:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761526/real 1789761526] req@ffff8dd211834700 x1876699496044032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761526.667585:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496044032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761526.667596:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761526.667600:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761526.667603:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496044032:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761526.667622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496044160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761526.667642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761526.667661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761526/real 1789761526] req@ffff8dd211834000 x1876699496044160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761526.667669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496044160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761526.667675:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761526.667677:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761526.667679:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496044160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761526.667683:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496044288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761526.667696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761526.667703:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761526/real 1789761526] req@ffff8dd211836a00 x1876699496044288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761526.667710:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496044288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761526.667719:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761526.667722:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761526.667724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496044288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761526.667727:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496044416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761526.667754:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761526.667762:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761526/real 1789761526] req@ffff8dd211835c00 x1876699496044416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761526.667770:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496044416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761526.667776:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761526.667778:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761526.667780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496044416:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761527.243309:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761528.267357:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761529.291323:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761530.315360:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761531.339325:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761531.787308:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761531.787318:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761531.787322:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761531.787333:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1511 00000100:00080000:3.0:1789761531.787345:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761531.787352:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761531.787391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496044544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761531.787414:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761531.787418:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761531.787419:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761531.787423:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1511 00000100:00080000:3.0:1789761531.787425:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761531.787427:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761531.787437:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761531.787447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496044672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761531.787457:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761531.787459:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761531.787461:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761531.787463:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1511 00000100:00080000:3.0:1789761531.787465:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761531.787467:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761531.787473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761531.787481:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496044544:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761531.787484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496044800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761531.787504:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761531.787507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761531.787508:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761531.787511:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1511 00000100:00080000:3.0:1789761531.787512:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761531.787515:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761531.787518:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761531.787526:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496044928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761531.787566:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761531.787603:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761531/real 1789761531] req@ffff8dd203147800 x1876699496044544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761531.787616:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496044544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761531.787624:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761531.787628:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761531.787630:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496044544:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761531.787654:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496044672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761531.787669:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761531.787677:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761531/real 1789761531] req@ffff8dd203147480 x1876699496044672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761531.787686:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496044672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761531.787691:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761531.787693:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761531.787695:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496044672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761531.787714:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496044800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761531.787735:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761531.787743:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761531/real 1789761531] req@ffff8dd203144000 x1876699496044800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761531.787751:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496044800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761531.787757:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761531.787759:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761531.787778:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496044800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761531.787782:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496044928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761531.787795:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761531.787802:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761531/real 1789761531] req@ffff8dd203147100 x1876699496044928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761531.787810:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496044928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761531.787815:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761531.787817:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761531.787819:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496044928:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761532.363341:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761533.387299:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761534.411302:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761535.435295:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761536.459360:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761536.907321:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761536.907330:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761536.907333:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761536.907343:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1516 00000100:00080000:0.0:1789761536.907352:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761536.907358:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761536.907396:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496045056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761536.907427:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761536.907431:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761536.907434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761536.907438:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1516 00000100:00080000:0.0:1789761536.907442:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761536.907446:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761536.907456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761536.907468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496045184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761536.907477:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761536.907481:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789761536.907483:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496045056:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761536.907484:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761536.907491:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1516 00000100:00080000:0.0:1789761536.907496:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761536.907498:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761536.907502:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761536.907511:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496045312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761536.907518:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761536.907521:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761536.907522:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761536.907524:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1516 00000100:00080000:0.0:1789761536.907526:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761536.907528:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761536.907531:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761536.907537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496045440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761536.907557:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761536.907602:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761536/real 1789761536] req@ffff8dd211837b80 x1876699496045056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761536.907621:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496045056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761536.907633:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761536.907637:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761536.907642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496045056:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761536.907681:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496045184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761536.907706:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761536.907721:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761536/real 1789761536] req@ffff8dd211837100 x1876699496045184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761536.907735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496045184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761536.907753:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761536.907757:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761536.907761:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496045184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761536.907767:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496045312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761536.907789:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761536.907801:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761536/real 1789761536] req@ffff8dd211836300 x1876699496045312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761536.907814:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496045312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761536.907824:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761536.907828:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761536.907832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496045312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761536.907838:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496045440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761536.907858:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761536.907869:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761536/real 1789761536] req@ffff8dd211837800 x1876699496045440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761536.907882:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496045440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761536.907891:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761536.907895:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761536.907898:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496045440:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761537.483286:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761538.507356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761539.531290:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761540.555348:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761541.579302:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761542.027271:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761542.027280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761542.027283:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761542.027293:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1521 00000100:00080000:3.0:1789761542.027303:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761542.027310:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761542.027353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496045568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761542.027383:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761542.027386:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761542.027387:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761542.027390:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1521 00000100:00080000:3.0:1789761542.027394:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761542.027398:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761542.027406:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761542.027424:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496045568:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761542.027424:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496045696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761542.027439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761542.027441:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761542.027442:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761542.027445:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1521 00000100:00080000:3.0:1789761542.027447:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761542.027450:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761542.027453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761542.027462:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496045824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761542.027469:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761542.027471:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761542.027472:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761542.027475:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1521 00000100:00080000:3.0:1789761542.027477:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761542.027479:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761542.027482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761542.027489:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496045952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761542.027495:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761542.027538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761542/real 1789761542] req@ffff8dd203145880 x1876699496045568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761542.027550:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496045568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761542.027558:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761542.027561:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761542.027564:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496045568:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761542.027584:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496045696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761542.027601:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761542.027609:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761542/real 1789761542] req@ffff8dd203147100 x1876699496045696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761542.027617:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496045696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761542.027637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761542.027640:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761542.027642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496045696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761542.027646:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496045824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761542.027666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761542.027679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761542/real 1789761542] req@ffff8dd203144000 x1876699496045824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761542.027702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496045824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761542.027709:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761542.027711:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761542.027713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496045824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761542.027717:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496045952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761542.027733:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761542.027741:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761542/real 1789761542] req@ffff8dd203147480 x1876699496045952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761542.027748:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496045952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761542.027753:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761542.027755:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761542.027757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496045952:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761542.603333:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761543.627308:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761544.651332:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761545.675331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761546.699325:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761547.147259:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761547.147269:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761547.147272:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761547.147281:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1526 00000100:00080000:3.0:1789761547.147290:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761547.147296:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761547.147330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496046080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761547.147351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761547.147354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761547.147356:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761547.147358:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1526 00000100:00080000:3.0:1789761547.147361:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761547.147363:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761547.147370:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761547.147378:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496046208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761547.147385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761547.147388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761547.147389:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761547.147391:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1526 00000100:00080000:3.0:1789761547.147393:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761547.147398:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761547.147401:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496046080:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789761547.147403:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761547.147413:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496046336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761547.147421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761547.147424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761547.147425:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761547.147427:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1526 00000100:00080000:3.0:1789761547.147429:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761547.147432:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761547.147435:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761547.147445:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496046464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761547.147493:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496046208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761547.147529:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496046336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761547.147557:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496046464:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761547.723314:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761548.747337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761549.771337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761550.219427:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789761550.219469:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761550.219482:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761550.219518:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761550.219530:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761550.219541:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789761550.219551:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761547/real 1789761550] req@ffff8dd203145c00 x1876699496046336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761550.219592:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496046336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761550.219608:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761550.219613:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761550.219617:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496046336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761550.219626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761547/real 1789761550] req@ffff8dd203145180 x1876699496046464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761550.219639:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496046464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761550.219647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761550.219651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761550.219654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496046464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761550.219661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761547/real 1789761550] req@ffff8dd203147800 x1876699496046080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761550.219674:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496046080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761550.219682:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761550.219686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761550.219689:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496046080:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761550.219696:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761547/real 1789761550] req@ffff8dd203146300 x1876699496046208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761550.219709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496046208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761550.219719:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761550.219729:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761550.219733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496046208:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761550.795301:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761551.819357:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761552.267237:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761552.267249:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761552.267254:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761552.267267:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1531 00000100:00080000:3.0:1789761552.267281:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761552.267290:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761552.267328:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496046592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761552.267354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761552.267357:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761552.267358:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761552.267362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1531 00000100:00080000:3.0:1789761552.267367:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761552.267370:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761552.267380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761552.267389:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496046592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761552.267396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496046720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761552.267406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761552.267409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761552.267410:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761552.267414:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1531 00000100:00080000:3.0:1789761552.267419:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761552.267423:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761552.267429:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761552.267443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496046848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761552.267454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761552.267458:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761552.267460:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761552.267464:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1531 00000100:00080000:3.0:1789761552.267469:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761552.267475:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761552.267476:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761552.267485:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761552.267499:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496046976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761552.267538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761552/real 1789761552] req@ffff8dd203145f80 x1876699496046592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761552.267575:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496046592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761552.267588:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761552.267593:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761552.267598:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496046592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761552.267631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496046720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761552.267660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761552.267675:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761552/real 1789761552] req@ffff8dd203144380 x1876699496046720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761552.267687:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496046720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761552.267696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761552.267699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761552.267703:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496046720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761552.267709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496046848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761552.267731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761552.267744:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761552/real 1789761552] req@ffff8dd203144a80 x1876699496046848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761552.267756:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496046848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761552.267765:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761552.267769:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761552.267773:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496046848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761552.267779:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496046976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761552.267801:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761552.267813:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761552/real 1789761552] req@ffff8dd203144700 x1876699496046976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761552.267827:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496046976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761552.267837:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761552.267840:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761552.267844:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496046976:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761552.843344:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761553.867335:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761554.891299:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761555.915337:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761556.939297:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761557.387237:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761557.387246:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761557.387249:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761557.387259:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1536 00000100:00080000:3.0:1789761557.387268:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761557.387274:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761557.387304:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496047104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761557.387326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761557.387329:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761557.387331:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761557.387333:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1536 00000100:00080000:3.0:1789761557.387336:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761557.387338:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761557.387346:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761557.387355:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496047232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761557.387363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761557.387366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761557.387367:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761557.387371:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496047104:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761557.387373:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1536 00000100:00080000:3.0:1789761557.387378:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761557.387381:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761557.387386:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761557.387395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496047360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761557.387401:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761557.387404:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761557.387405:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761557.387408:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1536 00000100:00080000:3.0:1789761557.387410:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761557.387412:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761557.387415:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761557.387422:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496047488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761557.387435:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761557.387475:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761557/real 1789761557] req@ffff8dd203145500 x1876699496047104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761557.387489:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496047104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761557.387496:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761557.387499:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761557.387502:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496047104:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761557.387536:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496047232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761557.387554:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761557.387564:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761557/real 1789761557] req@ffff8dd203146a00 x1876699496047232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761557.387572:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496047232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761557.387578:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761557.387580:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761557.387583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496047232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761557.387586:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496047360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761557.387598:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761557.387604:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761557/real 1789761557] req@ffff8dd203144e00 x1876699496047360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761557.387612:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496047360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761557.387617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761557.387619:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761557.387621:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496047360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761557.387624:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496047488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761557.387634:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761557.387640:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761557/real 1789761557] req@ffff8dd203146d80 x1876699496047488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761557.387647:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496047488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761557.387652:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761557.387653:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761557.387655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496047488:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761557.963323:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761558.987290:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761560.011356:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761561.035314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761562.059422:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761562.507299:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761562.507308:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761562.507311:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761562.507320:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1542 00000100:00080000:1.0:1789761562.507329:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761562.507335:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761562.507366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496047616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761562.507393:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761562.507396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761562.507398:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761562.507403:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1542 00000100:00080000:1.0:1789761562.507408:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761562.507413:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761562.507423:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761562.507430:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496047616:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761562.507443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496047744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761562.507453:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761562.507455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761562.507457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761562.507460:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1542 00000100:00080000:1.0:1789761562.507493:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761562.507497:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761562.507502:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761562.507506:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761562.507515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496047872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761562.507525:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761562.507528:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761562.507530:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761562.507532:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1542 00000100:00080000:1.0:1789761562.507535:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761562.507538:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761562.507542:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761562.507550:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496048000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761562.507554:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761562/real 1789761562] req@ffff8dd20504d180 x1876699496047616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761562.507588:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496047616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761562.507597:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761562.507601:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761562.507604:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496047616:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761562.507624:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496047744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761562.507642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761562.507651:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761562/real 1789761562] req@ffff8dd20504e680 x1876699496047744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761562.507659:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496047744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761562.507665:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761562.507667:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761562.507669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496047744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761562.507672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496047872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761562.507683:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761562.507689:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761562/real 1789761562] req@ffff8dd20504c700 x1876699496047872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761562.507696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496047872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761562.507706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761562.507708:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761562.507710:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496047872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761562.507714:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496048000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761562.507723:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761562.507729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761562/real 1789761562] req@ffff8dd20504e300 x1876699496048000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761562.507741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496048000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761562.507747:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761562.507749:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761562.507751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496048000:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761563.083298:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761564.107310:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761565.131364:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761566.155329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761567.179364:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761567.627302:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761567.627316:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761567.627322:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761567.627334:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1547 00000100:00080000:1.0:1789761567.627346:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761567.627355:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761567.627405:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496048128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761567.627443:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761567.627448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761567.627451:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761567.627456:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1547 00000100:00080000:1.0:1789761567.627461:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761567.627469:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496048128:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761567.627475:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761567.627487:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761567.627505:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496048256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761567.627518:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761567.627523:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761567.627525:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761567.627527:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761567.627533:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1547 00000100:00080000:1.0:1789761567.627540:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761567.627546:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761567.627552:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761567.627570:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761567/real 1789761567] req@ffff8dd20504d500 x1876699496048128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761567.627570:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496048384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761567.627604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496048128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761567.627604:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761567.627615:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761567.627620:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761567.627622:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761567.627629:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1789761567.627629:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1547 00000100:00100000:0.0:1789761567.627635:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496048128:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761567.627637:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761567.627644:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761567.627651:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761567.627658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496048256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761567.627668:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496048512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761567.627683:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761567.627695:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761567/real 1789761567] req@ffff8dd20504df80 x1876699496048256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761567.627704:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496048256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761567.627711:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761567.627713:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761567.627715:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496048256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761567.627719:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496048384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761567.627730:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761567.627736:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761567/real 1789761567] req@ffff8dd20504fb80 x1876699496048384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761567.627744:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496048384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761567.627749:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761567.627751:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761567.627753:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496048384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761567.627761:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496048512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761567.627773:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761567.627779:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761567/real 1789761567] req@ffff8dd20504f800 x1876699496048512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761567.627786:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496048512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761567.627791:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761567.627793:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761567.627795:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496048512:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761568.203332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761569.227291:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761570.251315:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761571.275311:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761572.299288:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761572.747258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761572.747268:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761572.747271:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761572.747280:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1552 00000100:00080000:2.0:1789761572.747288:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761572.747294:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761572.747326:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496048640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761572.747346:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761572.747349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761572.747350:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761572.747353:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1552 00000100:00080000:2.0:1789761572.747355:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761572.747357:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761572.747365:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761572.747376:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496048768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761572.747383:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761572.747385:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761572.747387:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761572.747389:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1552 00000100:00080000:2.0:1789761572.747402:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761572.747406:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761572.747414:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761572.747417:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496048640:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761572.747433:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496048896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761572.747448:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761572.747453:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761572.747455:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761572.747470:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1552 00000100:00080000:2.0:1789761572.747474:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761572.747478:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761572.747482:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761572.747493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496049024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761572.747537:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761572.747591:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761572/real 1789761572] req@ffff8dd205264700 x1876699496048640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761572.747608:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496048640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761572.747618:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761572.747622:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761572.747625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496048640:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761572.747632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496048768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761572.747652:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761572.747661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761572/real 1789761572] req@ffff8dd205266a00 x1876699496048768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761572.747676:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496048768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761572.747682:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761572.747684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761572.747686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496048768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761572.747709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496048896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761572.747723:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761572.747730:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761572/real 1789761572] req@ffff8dd205265880 x1876699496048896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761572.747739:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496048896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761572.747745:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761572.747747:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761572.747749:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496048896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761572.747753:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496049024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761572.747763:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761572.747769:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761572/real 1789761572] req@ffff8dd205264000 x1876699496049024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761572.747776:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496049024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761572.747786:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761572.747789:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761572.747791:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496049024:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761573.323322:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761574.347369:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761575.371321:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761576.395328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761577.419323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761577.867281:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761577.867295:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761577.867299:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761577.867311:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1557 00000100:00080000:0.0:1789761577.867321:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761577.867328:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761577.867375:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496049152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761577.867398:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761577.867401:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761577.867402:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761577.867406:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1557 00000100:00080000:0.0:1789761577.867408:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761577.867411:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761577.867419:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761577.867428:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496049280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761577.867436:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761577.867439:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761577.867441:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761577.867443:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496049152:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761577.867445:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1557 00000100:00080000:0.0:1789761577.867450:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761577.867453:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761577.867457:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761577.867466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496049408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761577.867473:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761577.867476:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761577.867477:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761577.867479:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1557 00000100:00080000:0.0:1789761577.867495:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761577.867497:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761577.867500:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761577.867505:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761577.867509:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496049536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761577.867545:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761577/real 1789761577] req@ffff8dd211837800 x1876699496049152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761577.867559:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496049152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761577.867571:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761577.867574:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761577.867577:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496049152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761577.867596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496049280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761577.867611:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761577.867620:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761577/real 1789761577] req@ffff8dd211836300 x1876699496049280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761577.867628:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496049280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761577.867637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761577.867640:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761577.867644:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496049280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761577.867650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496049408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761577.867670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761577.867688:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761577/real 1789761577] req@ffff8dd211837100 x1876699496049408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761577.867697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496049408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761577.867703:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761577.867705:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761577.867707:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496049408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761577.867712:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496049536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761577.867726:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761577.867733:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761577/real 1789761577] req@ffff8dd211837b80 x1876699496049536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761577.867741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496049536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761577.867746:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761577.867747:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761577.867749:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496049536:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761578.443311:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761579.467320:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761580.491307:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761581.515337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761582.539341:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761582.987267:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761582.987276:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761582.987280:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761582.987289:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1562 00000100:00080000:2.0:1789761582.987298:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761582.987304:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761582.987354:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496049664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761582.987388:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761582.987391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761582.987393:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761582.987396:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1562 00000100:00080000:2.0:1789761582.987399:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761582.987402:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761582.987410:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761582.987422:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496049792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761582.987430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761582.987433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761582.987434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761582.987438:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496049664:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761582.987440:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1562 00000100:00080000:2.0:1789761582.987446:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761582.987450:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761582.987454:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761582.987465:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496049920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761582.987472:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761582.987475:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761582.987476:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761582.987479:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1562 00000100:00080000:2.0:1789761582.987482:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761582.987484:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761582.987487:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761582.987496:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496050048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761582.987503:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761582.987542:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761582/real 1789761582] req@ffff8dd205267b80 x1876699496049664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761582.987555:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496049664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761582.987563:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761582.987566:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761582.987568:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496049664:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761582.987587:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496049792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761582.987601:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761582.987612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761582/real 1789761582] req@ffff8dd205264e00 x1876699496049792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761582.987625:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496049792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761582.987634:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761582.987637:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761582.987641:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496049792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761582.987648:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496049920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761582.987669:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761582.987679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761582/real 1789761582] req@ffff8dd205264380 x1876699496049920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761582.987687:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496049920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761582.987692:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761582.987694:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761582.987697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496049920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761582.987701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496050048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761582.987713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761582.987718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761582/real 1789761582] req@ffff8dd205267800 x1876699496050048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761582.987731:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496050048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761582.987741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761582.987743:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761582.987746:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496050048:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761583.563325:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761584.587346:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761585.611342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761586.635377:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761587.659381:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761588.107261:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761588.107270:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761588.107285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761588.107298:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1567 00000100:00080000:2.0:1789761588.107310:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761588.107317:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761588.107361:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496050176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761588.107384:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761588.107387:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761588.107389:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761588.107392:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1567 00000100:00080000:2.0:1789761588.107394:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761588.107397:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761588.107407:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761588.107417:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496050304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761588.107430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761588.107433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761588.107434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761588.107437:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1567 00000100:00080000:2.0:1789761588.107439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761588.107442:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761588.107446:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761588.107453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496050432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761588.107464:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496050176:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761588.107467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761588.107477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761588.107482:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761588.107487:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1567 00000100:00080000:2.0:1789761588.107489:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761588.107492:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761588.107496:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761588.107506:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496050560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761588.107544:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761588.107584:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761588/real 1789761588] req@ffff8dd205267100 x1876699496050176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761588.107597:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496050176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761588.107606:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761588.107609:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761588.107612:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496050176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761588.107617:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496050304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761588.107632:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761588.107639:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761588/real 1789761588] req@ffff8dd205266680 x1876699496050304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761588.107647:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496050304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761588.107652:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761588.107654:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761588.107656:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496050304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761588.107680:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496050432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761588.107691:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761588.107698:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761588/real 1789761588] req@ffff8dd205265500 x1876699496050432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761588.107705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496050432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761588.107710:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761588.107712:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761588.107714:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496050432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761588.107718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496050560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761588.107728:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761588.107734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761588/real 1789761588] req@ffff8dd205264a80 x1876699496050560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761588.107741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496050560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761588.107746:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761588.107748:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761588.107750:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496050560:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761588.683366:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761589.707323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761590.731335:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761591.755325:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761592.779349:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761593.227305:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761593.227318:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761593.227324:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761593.227336:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1572 00000100:00080000:0.0:1789761593.227350:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761593.227360:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761593.227426:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496050688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761593.227467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761593.227472:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761593.227475:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761593.227481:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1572 00000100:00080000:0.0:1789761593.227486:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761593.227493:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761593.227505:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761593.227521:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496050816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761593.227537:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761593.227538:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496050688:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761593.227547:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761593.227551:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761593.227556:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1572 00000100:00080000:0.0:1789761593.227571:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761593.227576:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761593.227583:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761593.227599:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496050944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761593.227613:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761593.227617:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761593.227619:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761593.227637:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1572 00000100:00080000:0.0:1789761593.227641:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761593.227646:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761593.227652:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761593.227665:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496051072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761593.227734:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761593.227795:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761593/real 1789761593] req@ffff8dd211835c00 x1876699496050688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761593.227818:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496050688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761593.227827:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761593.227831:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761593.227834:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496050688:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761593.227860:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496050816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761593.227880:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761593.227890:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761593/real 1789761593] req@ffff8dd211836a00 x1876699496050816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761593.227898:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496050816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761593.227904:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761593.227907:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761593.227909:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496050816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761593.227913:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496050944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761593.227932:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761593.227943:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761593/real 1789761593] req@ffff8dd211834000 x1876699496050944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761593.227955:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496050944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761593.227970:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761593.227985:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761593.227989:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496050944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761593.228004:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496051072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761593.228026:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761593.228038:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761593/real 1789761593] req@ffff8dd211834700 x1876699496051072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761593.228047:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496051072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761593.228053:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761593.228055:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761593.228057:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496051072:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761593.803353:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761594.827342:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761595.851328:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761596.875351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761597.899303:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761598.347289:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761598.347302:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761598.347306:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761598.347317:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1577 00000100:00080000:0.0:1789761598.347328:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761598.347336:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761598.347373:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496051200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761598.347401:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761598.347405:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761598.347407:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761598.347411:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1577 00000100:00080000:0.0:1789761598.347415:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761598.347420:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761598.347429:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761598.347442:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496051328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761598.347456:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761598.347456:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496051200:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761598.347464:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761598.347468:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761598.347473:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1577 00000100:00080000:0.0:1789761598.347476:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761598.347488:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761598.347493:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761598.347510:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496051456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761598.347521:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761598.347525:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761598.347527:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761598.347533:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761598.347536:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1577 00000100:00080000:0.0:1789761598.347542:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761598.347547:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761598.347553:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761598.347565:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496051584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761598.347585:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761598/real 1789761598] req@ffff8dd211835f80 x1876699496051200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761598.347608:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496051200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761598.347624:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761598.347627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761598.347630:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496051200:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761598.347651:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496051328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761598.347671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761598.347681:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761598/real 1789761598] req@ffff8dd211834380 x1876699496051328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761598.347690:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496051328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761598.347696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761598.347698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761598.347701:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496051328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761598.347704:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496051456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761598.347717:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761598.347723:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761598/real 1789761598] req@ffff8dd211836680 x1876699496051456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761598.347731:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496051456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761598.347737:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761598.347739:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761598.347741:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496051456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761598.347744:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496051584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761598.347764:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761598.347771:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761598/real 1789761598] req@ffff8dd211834a80 x1876699496051584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761598.347779:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496051584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761598.347785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761598.347787:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761598.347789:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496051584:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761598.923346:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761599.947319:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761600.971347:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761601.995304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761603.019338:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761603.467301:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761603.467311:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761603.467315:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761603.467323:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1583 00000100:00080000:0.0:1789761603.467332:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761603.467338:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761603.467371:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496051712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761603.467394:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761603.467396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761603.467398:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761603.467400:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1583 00000100:00080000:0.0:1789761603.467402:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761603.467405:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761603.467413:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761603.467427:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496051840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761603.467439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761603.467441:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496051712:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761603.467447:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761603.467451:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761603.467456:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1583 00000100:00080000:0.0:1789761603.467462:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761603.467467:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761603.467473:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761603.467486:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496051968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761603.467494:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761603.467497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761603.467498:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761603.467502:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789761603.467506:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1583 00000100:00080000:0.0:1789761603.467510:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761603.467512:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761603.467516:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761603.467529:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496052096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761603.467553:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761603/real 1789761603] req@ffff8dd211835180 x1876699496051712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761603.467568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496051712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761603.467576:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761603.467579:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761603.467582:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496051712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761603.467602:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496051840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761603.467618:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761603.467629:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761603/real 1789761603] req@ffff8dd211837480 x1876699496051840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761603.467642:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496051840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761603.467658:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761603.467662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761603.467666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496051840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761603.467672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496051968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761603.467690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761603.467699:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761603/real 1789761603] req@ffff8dd211834e00 x1876699496051968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761603.467707:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496051968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761603.467712:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761603.467714:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761603.467716:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496051968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761603.467719:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496052096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761603.467730:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761603.467736:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761603/real 1789761603] req@ffff8dd211836d80 x1876699496052096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761603.467742:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496052096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761603.467747:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761603.467749:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761603.467751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496052096:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761604.043397:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761605.067334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761606.091349:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761607.115316:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761608.139356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761608.587297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761608.587307:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761608.587310:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761608.587319:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1588 00000100:00080000:3.0:1789761608.587328:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761608.587333:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761608.587370:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496052224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761608.587401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761608.587404:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761608.587407:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761608.587412:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1588 00000100:00080000:3.0:1789761608.587416:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761608.587419:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761608.587427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761608.587440:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496052352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761608.587450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761608.587453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761608.587454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761608.587459:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1588 00000100:00100000:1.0:1789761608.587461:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496052224:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761608.587462:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761608.587469:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761608.587475:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761608.587484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496052480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761608.587492:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761608.587494:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761608.587496:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761608.587498:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1588 00000100:00080000:3.0:1789761608.587500:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761608.587503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761608.587506:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761608.587513:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496052608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761608.587531:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761608.587582:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761608/real 1789761608] req@ffff8dd203147b80 x1876699496052224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761608.587601:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496052224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761608.587612:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761608.587616:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761608.587620:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496052224:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761608.587642:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496052352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761608.587665:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761608.587678:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761608/real 1789761608] req@ffff8dd203146d80 x1876699496052352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761608.587691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496052352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761608.587700:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761608.587703:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761608.587707:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496052352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761608.587713:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496052480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761608.587743:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761608.587755:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761608/real 1789761608] req@ffff8dd203144e00 x1876699496052480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761608.587768:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496052480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761608.587777:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761608.587780:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761608.587784:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496052480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761608.587790:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496052608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761608.587807:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761608.587817:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761608/real 1789761608] req@ffff8dd203146a00 x1876699496052608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761608.587827:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496052608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761608.587835:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761608.587838:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761608.587841:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496052608:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761609.163324:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761610.187331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761611.211416:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761612.235343:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761613.259319:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761613.707278:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761613.707288:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761613.707291:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761613.707300:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1593 00000100:00080000:2.0:1789761613.707310:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761613.707316:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761613.707350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496052736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761613.707373:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761613.707376:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761613.707377:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761613.707380:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1593 00000100:00080000:2.0:1789761613.707382:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761613.707384:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761613.707397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761613.707412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496052864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761613.707424:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761613.707428:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761613.707431:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761613.707436:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1593 00000100:00080000:2.0:1789761613.707440:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761613.707445:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496052736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761613.707448:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761613.707457:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761613.707473:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496052992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761613.707486:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761613.707491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761613.707493:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761613.707497:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1593 00000100:00080000:2.0:1789761613.707502:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761613.707506:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761613.707512:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761613.707526:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496053120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761613.707558:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496052864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761613.707606:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496052992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761613.707636:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496053120:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761614.283274:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761615.307305:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761616.331318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761616.779470:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789761616.779519:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761616.779538:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761616.779592:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761616.779604:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761616.779616:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789761616.779668:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761613/real 1789761616] req@ffff8dd205265500 x1876699496052992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761616.779703:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496052992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761616.779716:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761616.779721:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761616.779725:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496052992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761616.779735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761613/real 1789761616] req@ffff8dd205266680 x1876699496053120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761616.779748:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496053120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761616.779757:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761616.779760:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761616.779764:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496053120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761616.779772:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761613/real 1789761616] req@ffff8dd205267480 x1876699496052736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761616.779784:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496052736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761616.779793:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761616.779797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761616.779800:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496052736:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761616.779808:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761613/real 1789761616] req@ffff8dd205264a80 x1876699496052864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761616.779821:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496052864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761616.779829:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761616.779832:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761616.779836:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496052864:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761617.355291:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761618.379336:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761618.827326:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761618.827339:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761618.827344:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761618.827358:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1598 00000100:00080000:2.0:1789761618.827373:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761618.827382:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761618.827435:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496053248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761618.827471:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761618.827475:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761618.827477:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761618.827482:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1598 00000100:00080000:2.0:1789761618.827487:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761618.827491:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761618.827504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761618.827526:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496053376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761618.827532:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496053248:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761618.827546:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761618.827551:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761618.827554:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761618.827559:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1598 00000100:00080000:2.0:1789761618.827563:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761618.827568:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761618.827575:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761618.827594:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496053504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761618.827608:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761618.827613:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761618.827615:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761618.827620:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1598 00000100:00080000:2.0:1789761618.827624:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761618.827626:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761618.827633:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761618.827640:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761618.827655:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496053632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761618.827685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761618/real 1789761618] req@ffff8dd205267100 x1876699496053248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761618.827709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496053248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761618.827723:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761618.827728:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761618.827732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496053248:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761618.827762:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496053376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761618.827790:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761618.827806:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761618/real 1789761618] req@ffff8dd205267800 x1876699496053376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761618.827820:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496053376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761618.827827:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761618.827830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761618.827833:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496053376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761618.827839:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496053504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761618.827864:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761618.827878:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761618/real 1789761618] req@ffff8dd205264380 x1876699496053504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761618.827891:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496053504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761618.827900:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761618.827904:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761618.827908:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496053504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761618.827931:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496053632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761618.827956:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761618.827969:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761618/real 1789761618] req@ffff8dd205264e00 x1876699496053632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761618.827982:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496053632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761618.827990:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761618.827994:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761618.827999:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496053632:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761619.403377:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761620.427318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761621.451343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761622.475401:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761623.499399:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761623.947295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761623.947305:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761623.947309:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761623.947321:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1603 00000100:00080000:2.0:1789761623.947333:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761623.947342:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761623.947390:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496053760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761623.947422:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761623.947427:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761623.947429:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761623.947434:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1603 00000100:00080000:2.0:1789761623.947439:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761623.947444:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761623.947456:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761623.947474:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496053888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761623.947483:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496053760:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761623.947491:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761623.947496:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761623.947498:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761623.947502:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1603 00000100:00080000:2.0:1789761623.947505:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761623.947509:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761623.947514:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761623.947525:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496054016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761623.947532:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761623.947535:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761623.947536:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761623.947539:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1603 00000100:00080000:2.0:1789761623.947542:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761623.947545:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761623.947549:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761623.947556:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496054144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761623.947578:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761623.947626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761623/real 1789761623] req@ffff8dd205267b80 x1876699496053760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761623.947647:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496053760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761623.947659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761623.947664:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761623.947669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496053760:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761623.947693:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496053888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761623.947712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761623.947722:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761623/real 1789761623] req@ffff8dd205264000 x1876699496053888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761623.947730:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496053888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761623.947739:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761623.947742:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761623.947746:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496053888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761623.947753:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496054016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761623.947774:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761623.947787:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761623/real 1789761623] req@ffff8dd205265880 x1876699496054016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761623.947799:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496054016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761623.947817:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761623.947821:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761623.947824:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496054016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761623.947832:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496054144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761623.947853:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761623.947863:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761623/real 1789761623] req@ffff8dd205266a00 x1876699496054144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761623.947871:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496054144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761623.947877:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761623.947879:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761623.947881:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496054144:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761624.523362:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761625.547295:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761626.571293:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761627.595325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761628.619316:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761629.067280:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761629.067293:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761629.067297:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761629.067309:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1608 00000100:00080000:3.0:1789761629.067321:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761629.067328:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761629.067369:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496054272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761629.067398:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761629.067402:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761629.067403:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761629.067407:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1608 00000100:00080000:3.0:1789761629.067410:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761629.067412:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761629.067421:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761629.067432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496054400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761629.067446:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761629.067449:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761629.067451:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761629.067454:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1608 00000100:00080000:3.0:1789761629.067457:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761629.067459:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761629.067464:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496054272:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789761629.067464:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761629.067475:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496054528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761629.067483:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761629.067486:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761629.067487:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761629.067489:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1608 00000100:00080000:3.0:1789761629.067492:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761629.067494:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761629.067498:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761629.067506:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496054656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761629.067526:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761629.067578:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761629/real 1789761629] req@ffff8dd203145500 x1876699496054272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761629.067593:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496054272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761629.067602:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761629.067604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761629.067607:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496054272:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761629.067627:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496054400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761629.067652:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761629.067666:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761629/real 1789761629] req@ffff8dd203144700 x1876699496054400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761629.067679:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496054400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761629.067689:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761629.067692:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761629.067696:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496054400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761629.067703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496054528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761629.067725:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761629.067738:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761629/real 1789761629] req@ffff8dd203144a80 x1876699496054528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761629.067751:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496054528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761629.067760:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761629.067764:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761629.067768:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496054528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761629.067775:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496054656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761629.067793:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761629.067804:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761629/real 1789761629] req@ffff8dd203144380 x1876699496054656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761629.067816:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496054656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761629.067825:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761629.067828:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761629.067831:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496054656:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761629.643325:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761630.667299:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761631.691336:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761632.715308:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761633.739351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761634.187284:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761634.187294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761634.187298:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761634.187306:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1613 00000100:00080000:3.0:1789761634.187317:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761634.187325:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761634.187366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496054784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761634.187391:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761634.187394:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761634.187396:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761634.187399:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1613 00000100:00080000:3.0:1789761634.187402:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761634.187404:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761634.187412:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761634.187423:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496054912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761634.187432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761634.187433:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496054784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761634.187435:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761634.187439:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761634.187443:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1613 00000100:00080000:3.0:1789761634.187446:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761634.187448:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761634.187452:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761634.187460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496055040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761634.187467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761634.187470:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761634.187472:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761634.187474:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1613 00000100:00080000:3.0:1789761634.187477:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761634.187479:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761634.187482:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761634.187489:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496055168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761634.187494:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761634.187540:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761634/real 1789761634] req@ffff8dd203145f80 x1876699496054784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761634.187554:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496054784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761634.187562:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761634.187565:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761634.187567:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496054784:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761634.187586:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496054912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761634.187602:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761634.187610:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761634/real 1789761634] req@ffff8dd203146300 x1876699496054912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761634.187619:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496054912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761634.187625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761634.187627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761634.187629:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496054912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761634.187632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496055040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761634.187643:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761634.187649:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761634/real 1789761634] req@ffff8dd203147800 x1876699496055040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761634.187656:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496055040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761634.187662:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761634.187664:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761634.187666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496055040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761634.187669:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496055168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761634.187678:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761634.187684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761634/real 1789761634] req@ffff8dd203145180 x1876699496055168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761634.187692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496055168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761634.187701:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761634.187703:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761634.187705:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496055168:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761634.763298:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761635.787337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761636.811332:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761637.835348:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761638.859288:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761639.307242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761639.307255:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761639.307260:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761639.307271:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1618 00000100:00080000:3.0:1789761639.307281:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761639.307287:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761639.307321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496055296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761639.307347:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761639.307349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761639.307351:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761639.307354:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1618 00000100:00080000:3.0:1789761639.307356:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761639.307358:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761639.307366:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761639.307380:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496055424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761639.307391:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496055296:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761639.307404:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761639.307418:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761639.307420:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761639.307424:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1618 00000100:00080000:3.0:1789761639.307427:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761639.307430:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761639.307435:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761639.307445:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496055552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761639.307453:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761639.307456:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761639.307457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761639.307462:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1618 00000800:00000100:1.0:1789761639.307466:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761639.307467:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761639.307472:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761639.307476:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761639.307484:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496055680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761639.307517:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761639/real 1789761639] req@ffff8dd203145c00 x1876699496055296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761639.307537:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496055296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761639.307547:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761639.307550:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761639.307553:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496055296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761639.307578:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496055424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761639.307597:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761639.307607:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761639/real 1789761639] req@ffff8dd203147480 x1876699496055424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761639.307615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496055424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761639.307621:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761639.307623:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761639.307626:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496055424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761639.307630:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496055552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761639.307642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761639.307655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761639/real 1789761639] req@ffff8dd203144000 x1876699496055552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761639.307663:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496055552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761639.307669:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761639.307671:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761639.307673:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496055552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761639.307676:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496055680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761639.307688:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761639.307694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761639/real 1789761639] req@ffff8dd203147100 x1876699496055680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761639.307701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496055680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761639.307711:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761639.307713:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761639.307715:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496055680:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761639.883343:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761640.907332:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761641.931355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761642.955359:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761643.979351:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761644.427274:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761644.427285:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761644.427288:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761644.427297:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1624 00000100:00080000:0.0:1789761644.427306:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761644.427312:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761644.427349:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496055808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761644.427379:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761644.427383:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761644.427386:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761644.427390:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1624 00000100:00080000:0.0:1789761644.427395:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761644.427399:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761644.427408:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761644.427423:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496055936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761644.427423:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496055808:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761644.427438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761644.427441:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761644.427442:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761644.427446:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1624 00000100:00080000:0.0:1789761644.427448:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761644.427451:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761644.427455:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761644.427463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496056064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761644.427471:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761644.427474:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761644.427475:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761644.427477:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1624 00000100:00080000:0.0:1789761644.427480:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761644.427486:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761644.427486:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789761644.427493:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761644.427503:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496056192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761644.427536:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761644/real 1789761644] req@ffff8dd211836d80 x1876699496055808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761644.427551:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496055808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761644.427558:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761644.427561:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761644.427564:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496055808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761644.427584:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496055936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761644.427600:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761644.427609:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761644/real 1789761644] req@ffff8dd211834e00 x1876699496055936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761644.427617:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496055936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761644.427623:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761644.427625:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761644.427628:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496055936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761644.427631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496056064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761644.427642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761644.427648:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761644/real 1789761644] req@ffff8dd211837480 x1876699496056064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761644.427655:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496056064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761644.427665:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761644.427667:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761644.427669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496056064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761644.427673:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496056192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761644.427682:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761644.427692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761644/real 1789761644] req@ffff8dd211835180 x1876699496056192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761644.427704:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496056192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761644.427713:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761644.427716:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761644.427720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496056192:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761645.003359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761646.027309:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761647.051350:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761648.075355:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761649.099374:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761649.547235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761649.547248:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761649.547253:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761649.547264:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1629 00000100:00080000:0.0:1789761649.547278:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761649.547287:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761649.547321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496056320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761649.547343:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761649.547346:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761649.547348:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761649.547351:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1629 00000100:00080000:0.0:1789761649.547354:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761649.547357:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761649.547365:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761649.547376:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496056448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761649.547376:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496056320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761649.547390:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761649.547392:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761649.547394:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761649.547396:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1629 00000100:00080000:0.0:1789761649.547398:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761649.547401:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761649.547404:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761649.547413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496056576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761649.547428:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761649.547432:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761649.547434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761649.547438:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1629 00000100:00080000:0.0:1789761649.547442:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761649.547449:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789761649.547449:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789761649.547458:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761649.547474:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496056704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761649.547490:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761649/real 1789761649] req@ffff8dd211834a80 x1876699496056320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761649.547506:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496056320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761649.547515:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761649.547518:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761649.547521:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496056320:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761649.547545:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496056448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761649.547562:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761649.547571:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761649/real 1789761649] req@ffff8dd211836680 x1876699496056448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761649.547579:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496056448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761649.547585:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761649.547587:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761649.547589:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496056448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761649.547593:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496056576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761649.547605:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761649.547611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761649/real 1789761649] req@ffff8dd211834380 x1876699496056576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761649.547617:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496056576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761649.547623:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761649.547625:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761649.547627:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496056576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761649.547631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496056704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761649.547641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761649.547646:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761649/real 1789761649] req@ffff8dd211835f80 x1876699496056704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761649.547654:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496056704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761649.547659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761649.547660:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761649.547662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496056704:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761650.123333:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761651.147345:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761652.171312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761653.195335:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761654.219333:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761654.667328:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761654.667338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761654.667343:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761654.667353:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1634 00000100:00080000:3.0:1789761654.667366:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761654.667374:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761654.667424:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496056832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761654.667455:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761654.667460:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761654.667462:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761654.667467:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1634 00000100:00080000:3.0:1789761654.667471:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761654.667474:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761654.667485:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761654.667502:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496056832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761654.667507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496056960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761654.667517:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761654.667520:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761654.667521:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761654.667524:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1634 00000100:00080000:3.0:1789761654.667527:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761654.667529:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761654.667533:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761654.667543:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496057088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761654.667551:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761654.667554:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761654.667555:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761654.667558:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1634 00000100:00080000:3.0:1789761654.667562:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761654.667564:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761654.667567:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761654.667571:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761654.667576:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496057216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761654.667615:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761654/real 1789761654] req@ffff8dd203145880 x1876699496056832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761654.667631:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496056832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761654.667640:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761654.667643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761654.667646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496056832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761654.667674:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496056960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761654.667700:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761654.667714:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761654/real 1789761654] req@ffff8dd203147100 x1876699496056960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761654.667727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496056960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761654.667743:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761654.667747:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761654.667751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496056960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761654.667758:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496057088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761654.667791:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761654.667802:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761654/real 1789761654] req@ffff8dd203144000 x1876699496057088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761654.667810:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496057088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761654.667816:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761654.667818:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761654.667821:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496057088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761654.667824:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496057216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761654.667838:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761654.667844:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761654/real 1789761654] req@ffff8dd203147480 x1876699496057216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761654.667852:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496057216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761654.667857:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761654.667859:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761654.667862:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496057216:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761655.243329:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761656.267312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761657.291316:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761658.315288:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761659.339286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761659.787273:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761659.787282:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761659.787286:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761659.787297:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1639 00000100:00080000:3.0:1789761659.787306:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761659.787312:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761659.787347:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496057344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761659.787387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761659.787391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761659.787393:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761659.787399:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1639 00000100:00080000:3.0:1789761659.787403:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761659.787407:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761659.787415:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761659.787429:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496057472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761659.787431:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496057344:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761659.787446:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761659.787448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761659.787449:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761659.787452:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1639 00000100:00080000:3.0:1789761659.787455:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761659.787457:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761659.787460:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761659.787467:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496057600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761659.787474:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761659.787477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761659.787478:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761659.787481:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1639 00000100:00080000:3.0:1789761659.787483:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761659.787485:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761659.787488:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761659.787495:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496057728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761659.787503:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761659.787538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761659/real 1789761659] req@ffff8dd203145c00 x1876699496057344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761659.787552:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496057344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761659.787560:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761659.787563:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761659.787566:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496057344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761659.787584:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496057472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761659.787598:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761659.787606:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761659/real 1789761659] req@ffff8dd203145180 x1876699496057472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761659.787615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496057472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761659.787621:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761659.787623:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761659.787625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496057472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761659.787629:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496057600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761659.787639:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761659.787646:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761659/real 1789761659] req@ffff8dd203147800 x1876699496057600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761659.787654:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496057600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761659.787659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761659.787661:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761659.787662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496057600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761659.787666:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496057728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761659.787675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761659.787681:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761659/real 1789761659] req@ffff8dd203146300 x1876699496057728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761659.787689:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496057728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761659.787694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761659.787695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761659.787697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496057728:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761660.363292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761661.387351:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761662.411336:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761663.435297:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761664.459352:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761664.907291:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761664.907301:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761664.907305:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761664.907314:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1644 00000100:00080000:3.0:1789761664.907325:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761664.907331:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761664.907362:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496057856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761664.907389:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761664.907393:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761664.907395:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761664.907399:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1644 00000100:00080000:3.0:1789761664.907403:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761664.907408:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761664.907419:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761664.907433:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496057856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761664.907439:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496057984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761664.907454:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761664.907458:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761664.907460:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761664.907465:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1644 00000100:00080000:3.0:1789761664.907470:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761664.907475:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761664.907481:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789761664.907491:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761664.907498:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496058112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761664.907512:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761664.907516:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761664.907518:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761664.907523:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1644 00000100:00080000:3.0:1789761664.907528:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761664.907531:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761664/real 1789761664] req@ffff8dd203145f80 x1876699496057856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761664.907536:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761664.907547:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761664.907555:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496057856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761664.907565:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761664.907565:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496058240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761664.907572:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761664.907580:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496057856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761664.907609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496057984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761664.907634:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761664.907643:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761664/real 1789761664] req@ffff8dd203144380 x1876699496057984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761664.907651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496057984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761664.907657:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761664.907663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761664.907665:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496057984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761664.907670:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496058112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761664.907682:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761664.907689:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761664/real 1789761664] req@ffff8dd203144a80 x1876699496058112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761664.907697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496058112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761664.907715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761664.907718:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761664.907720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496058112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761664.907725:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496058240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761664.907739:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761664.907747:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761664/real 1789761664] req@ffff8dd203144700 x1876699496058240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761664.907756:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496058240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761664.907765:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761664.907768:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761664.907772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496058240:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761665.483352:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761666.507359:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761667.531338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761668.555283:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761669.579337:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761670.027262:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761670.027275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761670.027279:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761670.027290:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1649 00000100:00080000:3.0:1789761670.027302:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761670.027310:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761670.027349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496058368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761670.027377:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761670.027380:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761670.027382:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761670.027385:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1649 00000100:00080000:3.0:1789761670.027389:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761670.027392:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761670.027400:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761670.027414:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496058368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761670.027414:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496058496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761670.027429:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761670.027431:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761670.027432:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761670.027434:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1649 00000100:00080000:3.0:1789761670.027437:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761670.027439:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761670.027442:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761670.027450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496058624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761670.027461:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761670.027464:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761670.027466:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761670.027470:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1649 00000100:00080000:3.0:1789761670.027475:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761670.027481:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761670.027482:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761670.027490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761670.027505:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496058752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761670.027526:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761670/real 1789761670] req@ffff8dd203145500 x1876699496058368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761670.027542:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496058368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761670.027552:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761670.027555:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761670.027558:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496058368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761670.027576:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496058496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761670.027599:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761670.027613:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761670/real 1789761670] req@ffff8dd203146a00 x1876699496058496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761670.027628:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496058496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761670.027637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761670.027639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761670.027642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496058496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761670.027647:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496058624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761670.027678:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761670.027688:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761670/real 1789761670] req@ffff8dd203144e00 x1876699496058624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761670.027697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496058624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761670.027702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761670.027704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761670.027706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496058624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761670.027710:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496058752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761670.027722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761670.027729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761670/real 1789761670] req@ffff8dd203146d80 x1876699496058752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761670.027740:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496058752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761670.027755:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761670.027759:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761670.027762:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496058752:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761670.603349:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761671.627343:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761672.651327:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761673.675300:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761674.699329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761675.147295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761675.147309:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761675.147313:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761675.147325:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1654 00000100:00080000:3.0:1789761675.147337:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761675.147345:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761675.147381:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496058880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761675.147415:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761675.147419:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761675.147421:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761675.147426:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1654 00000100:00080000:3.0:1789761675.147429:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761675.147433:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761675.147442:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761675.147457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496059008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761675.147470:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496058880:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761675.147471:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761675.147479:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761675.147482:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761675.147488:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1654 00000100:00080000:3.0:1789761675.147491:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761675.147500:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761675.147506:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761675.147518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496059136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761675.147529:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761675.147534:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789761675.147535:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761675.147538:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761675.147543:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1654 00000100:00080000:3.0:1789761675.147548:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761675.147552:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761675.147563:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761675.147576:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496059264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761675.147578:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761675/real 1789761675] req@ffff8dd203147b80 x1876699496058880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761675.147609:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496058880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761675.147617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761675.147620:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761675.147635:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496058880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761675.147656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496059008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761675.147680:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761675.147694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761675/real 1789761675] req@ffff8dd203146d80 x1876699496059008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761675.147709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496059008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761675.147718:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761675.147722:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761675.147724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496059008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761675.147729:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496059136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761675.147747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761675.147755:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761675/real 1789761675] req@ffff8dd203144e00 x1876699496059136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761675.147763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496059136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761675.147769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761675.147771:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761675.147773:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496059136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761675.147776:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496059264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761675.147786:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761675.147792:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761675/real 1789761675] req@ffff8dd203146a00 x1876699496059264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761675.147799:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496059264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761675.147805:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761675.147807:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761675.147808:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496059264:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761675.723307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761676.747324:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761677.771329:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761678.795326:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761679.819321:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761680.267272:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761680.267282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761680.267286:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761680.267295:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1659 00000100:00080000:0.0:1789761680.267304:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761680.267311:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761680.267348:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496059392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761680.267377:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761680.267381:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761680.267383:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761680.267388:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1659 00000100:00080000:0.0:1789761680.267392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761680.267397:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761680.267408:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761680.267431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496059520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761680.267445:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761680.267448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761680.267453:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761680.267453:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496059392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761680.267462:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1659 00000100:00080000:0.0:1789761680.267465:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761680.267469:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761680.267473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761680.267484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496059648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761680.267492:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761680.267495:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761680.267496:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761680.267499:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1659 00000100:00080000:0.0:1789761680.267507:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761680.267511:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761680.267517:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761680.267530:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496059776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761680.267563:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496059520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761680.267612:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496059648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761680.267634:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496059776:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761680.843324:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761681.867389:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761682.891314:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761683.339425:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789761683.339467:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761683.339485:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761683.339530:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761683.339541:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761683.339550:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789761683.339587:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761680/real 1789761683] req@ffff8dd211834000 x1876699496059520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761683.339618:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496059520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761683.339632:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761683.339639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761683.339644:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496059520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761683.339653:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761680/real 1789761683] req@ffff8dd211836a00 x1876699496059648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761683.339669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496059648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761683.339679:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761683.339683:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761683.339686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496059648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761683.339694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761680/real 1789761683] req@ffff8dd211835c00 x1876699496059776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761683.339709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496059776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761683.339725:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761683.339730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761683.339734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496059776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761683.339742:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761680/real 1789761683] req@ffff8dd211834700 x1876699496059392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761683.339763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496059392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761683.339772:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761683.339776:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761683.339780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496059392:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789761683.915360:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761684.939337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761685.387255:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761685.387266:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761685.387269:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761685.387278:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1664 00000100:00080000:2.0:1789761685.387289:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761685.387297:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761685.387343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496059904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761685.387376:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761685.387381:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761685.387383:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761685.387388:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1664 00000100:00080000:2.0:1789761685.387392:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761685.387396:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761685.387415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761685.387430:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496059904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761685.387437:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496060032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761685.387451:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761685.387455:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761685.387457:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761685.387461:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1664 00000100:00080000:2.0:1789761685.387467:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761685.387471:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761685.387477:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761685.387491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496060160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761685.387501:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761685.387506:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761685.387510:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761685.387514:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761685.387519:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1664 00000100:00080000:2.0:1789761685.387523:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761685.387528:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761685.387534:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761685.387560:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761685/real 1789761685] req@ffff8dd205264700 x1876699496059904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761685.387564:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496060288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761685.387598:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496059904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761685.387613:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761685.387618:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761685.387623:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496059904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761685.387648:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496060032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761685.387676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761685.387691:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761685/real 1789761685] req@ffff8dd205266300 x1876699496060032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761685.387705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496060032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761685.387714:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761685.387718:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761685.387721:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496060032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761685.387729:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496060160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761685.387751:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761685.387764:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761685/real 1789761685] req@ffff8dd205265180 x1876699496060160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761685.387776:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496060160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761685.387785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761685.387789:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761685.387792:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496060160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761685.387799:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496060288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761685.387821:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761685.387832:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761685/real 1789761685] req@ffff8dd205266d80 x1876699496060288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761685.387853:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496060288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761685.387874:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761685.387878:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761685.387881:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496060288:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761685.963356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761686.987319:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761688.011323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761689.035303:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761690.059317:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761690.507265:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761690.507276:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761690.507281:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761690.507291:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1670 00000100:00080000:0.0:1789761690.507303:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761690.507310:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761690.507349:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496060416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761690.507371:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761690.507375:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761690.507376:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761690.507380:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1670 00000100:00080000:0.0:1789761690.507382:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761690.507385:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761690.507395:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761690.507404:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496060544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761690.507418:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496060416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761690.507425:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761690.507430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761690.507431:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761690.507435:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1670 00000100:00080000:0.0:1789761690.507438:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761690.507440:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761690.507444:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761690.507453:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496060672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761690.507461:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761690.507464:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761690.507465:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761690.507467:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1670 00000100:00080000:0.0:1789761690.507478:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761690.507481:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761690.507483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761690.507489:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761690.507493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496060800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761690.507546:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761690/real 1789761690] req@ffff8dd211837b80 x1876699496060416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761690.507563:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496060416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761690.507572:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761690.507576:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761690.507580:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496060416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761690.507602:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496060544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761690.507621:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761690.507630:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761690/real 1789761690] req@ffff8dd211837100 x1876699496060544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761690.507641:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496060544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761690.507658:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761690.507662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761690.507666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496060544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761690.507673:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496060672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761690.507698:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761690.507709:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761690/real 1789761690] req@ffff8dd211836300 x1876699496060672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761690.507718:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496060672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761690.507724:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761690.507726:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761690.507729:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496060672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761690.507733:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496060800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761690.507747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761690.507753:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761690/real 1789761690] req@ffff8dd211837800 x1876699496060800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761690.507761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496060800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761690.507766:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761690.507767:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761690.507770:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496060800:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761691.083302:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761692.107329:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761693.131312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761694.155368:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761695.179298:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761695.627270:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761695.627280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761695.627284:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761695.627293:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1675 00000100:00080000:1.0:1789761695.627303:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761695.627309:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761695.627344:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496060928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761695.627367:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761695.627370:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761695.627371:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761695.627374:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1675 00000100:00080000:1.0:1789761695.627377:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761695.627379:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761695.627387:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761695.627395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496061056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761695.627403:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761695.627406:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761695.627408:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761695.627412:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496060928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761695.627413:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1675 00000100:00080000:1.0:1789761695.627418:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761695.627423:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761695.627426:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761695.627434:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496061184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761695.627440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761695.627442:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761695.627443:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761695.627447:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1675 00000100:00080000:1.0:1789761695.627451:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761695.627455:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761695.627461:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761695.627485:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496061312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761695.627499:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761695.627543:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761695/real 1789761695] req@ffff8dd20504ce00 x1876699496060928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761695.627563:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496060928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761695.627575:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761695.627579:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761695.627583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496060928:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761695.627614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496061056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761695.627636:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761695.627648:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761695/real 1789761695] req@ffff8dd20504f100 x1876699496061056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761695.627660:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496061056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761695.627669:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761695.627672:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761695.627676:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496061056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761695.627681:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496061184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761695.627699:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761695.627710:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761695/real 1789761695] req@ffff8dd20504ed80 x1876699496061184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761695.627723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496061184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761695.627732:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761695.627736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761695.627739:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496061184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761695.627745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496061312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761695.627767:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761695.627778:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761695/real 1789761695] req@ffff8dd20504d880 x1876699496061312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761695.627791:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496061312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761695.627799:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761695.627802:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761695.627806:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496061312:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761696.203314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761697.227299:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761698.251325:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761699.275311:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761700.299319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761700.747326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761700.747338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761700.747343:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761700.747362:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1680 00000100:00080000:3.0:1789761700.747374:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761700.747381:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761700.747420:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496061440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761700.747464:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761700.747468:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761700.747471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761700.747476:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1680 00000100:00080000:3.0:1789761700.747480:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761700.747484:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761700.747494:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761700.747510:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496061568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761700.747522:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496061440:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761700.747527:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761700.747533:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761700.747535:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761700.747540:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1680 00000100:00080000:3.0:1789761700.747544:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761700.747549:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761700.747555:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761700.747568:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496061696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761700.747581:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761700.747584:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761700.747586:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761700.747590:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1680 00000100:00080000:3.0:1789761700.747595:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761700.747598:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761700.747603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1789761700.747604:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761700.747619:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496061824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761700.747660:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761700/real 1789761700] req@ffff8dd203145500 x1876699496061440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761700.747676:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496061440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761700.747686:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761700.747690:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761700.747694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496061440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761700.747718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496061568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761700.747738:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761700.747751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761700/real 1789761700] req@ffff8dd203144700 x1876699496061568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761700.747765:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496061568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761700.747775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761700.747779:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761700.747783:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496061568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761700.747790:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496061696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761700.747811:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761700.747820:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761700/real 1789761700] req@ffff8dd203144a80 x1876699496061696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761700.747837:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496061696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761700.747843:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761700.747846:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761700.747848:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496061696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761700.747853:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496061824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761700.747867:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761700.747874:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761700/real 1789761700] req@ffff8dd203144380 x1876699496061824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761700.747882:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496061824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761700.747887:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761700.747889:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761700.747891:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496061824:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761701.323369:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761702.347343:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761703.371391:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761704.395341:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761705.419368:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761705.867322:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761705.867335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761705.867340:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761705.867351:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1685 00000100:00080000:1.0:1789761705.867363:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761705.867371:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761705.867421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496061952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761705.867444:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761705.867447:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761705.867449:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761705.867452:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1685 00000100:00080000:1.0:1789761705.867454:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761705.867457:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761705.867465:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761705.867474:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496062080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761705.867482:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761705.867485:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761705.867488:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761705.867490:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496061952:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761705.867495:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1685 00000100:00080000:1.0:1789761705.867500:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761705.867504:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761705.867509:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761705.867524:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496062208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761705.867536:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761705.867540:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761705.867542:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761705.867546:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1685 00000800:00000100:0.0:1789761705.867550:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761705.867551:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761705.867565:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761705.867572:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761705.867588:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496062336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761705.867591:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761705/real 1789761705] req@ffff8dd20504c380 x1876699496061952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761705.867623:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496061952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761705.867631:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761705.867634:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761705.867637:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496061952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761705.867658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496062080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761705.867673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761705.867682:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761705/real 1789761705] req@ffff8dd20504dc00 x1876699496062080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761705.867690:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496062080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761705.867696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761705.867699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761705.867701:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496062080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761705.867705:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496062208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761705.867715:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761705.867721:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761705/real 1789761705] req@ffff8dd20504c000 x1876699496062208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761705.867729:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496062208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761705.867734:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761705.867736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761705.867738:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496062208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761705.867741:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496062336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761705.867751:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761705.867757:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761705/real 1789761705] req@ffff8dd20504ea00 x1876699496062336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761705.867764:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496062336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761705.867770:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761705.867772:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761705.867774:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496062336:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761706.443349:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761707.467372:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761708.491385:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761709.515383:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761710.539303:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761710.987274:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761710.987286:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761710.987291:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761710.987303:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1690 00000100:00080000:1.0:1789761710.987315:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761710.987322:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761710.987373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496062464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761710.987402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761710.987406:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761710.987408:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761710.987413:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1690 00000100:00080000:1.0:1789761710.987417:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761710.987422:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761710.987433:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761710.987442:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496062464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761710.987452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496062592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761710.987462:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761710.987465:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761710.987466:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761710.987469:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1690 00000100:00080000:1.0:1789761710.987473:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761710.987477:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761710.987482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761710.987491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496062720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761710.987499:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761710.987501:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761710.987502:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761710.987504:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1690 00000100:00080000:1.0:1789761710.987507:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761710.987509:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761710.987515:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789761710.987518:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761710.987533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496062848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761710.987562:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761710/real 1789761710] req@ffff8dd20504ca80 x1876699496062464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761710.987581:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496062464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761710.987592:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761710.987596:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761710.987600:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496062464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761710.987621:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496062592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761710.987644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761710.987656:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761710/real 1789761710] req@ffff8dd20504ea00 x1876699496062592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761710.987669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496062592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761710.987677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761710.987681:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761710.987685:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496062592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761710.987690:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496062720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761710.987709:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761710.987720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761710/real 1789761710] req@ffff8dd20504c000 x1876699496062720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761710.987731:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496062720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761710.987739:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761710.987742:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761710.987745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496062720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761710.987751:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496062848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761710.987774:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761710.987784:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761710/real 1789761710] req@ffff8dd20504dc00 x1876699496062848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761710.987801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496062848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761710.987809:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761710.987812:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761710.987816:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496062848:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761711.563302:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761712.587283:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761713.611374:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761714.635300:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761715.659369:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761716.107346:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761716.107356:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761716.107359:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761716.107369:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1695 00000100:00080000:3.0:1789761716.107378:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761716.107384:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761716.107420:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496062976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761716.107442:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761716.107452:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761716.107458:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761716.107462:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1695 00000100:00080000:3.0:1789761716.107466:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761716.107470:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761716.107479:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761716.107495:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496063104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761716.107507:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761716.107508:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496062976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761716.107510:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761716.107514:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761716.107521:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1695 00000100:00080000:3.0:1789761716.107525:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761716.107528:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761716.107532:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761716.107542:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496063232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761716.107550:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761716.107553:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761716.107554:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761716.107556:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1695 00000100:00080000:3.0:1789761716.107559:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761716.107561:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761716.107564:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761716.107570:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496063360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761716.107581:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761716.107619:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761716/real 1789761716] req@ffff8dd203145f80 x1876699496062976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761716.107633:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496062976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761716.107641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761716.107643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761716.107647:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496062976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761716.107665:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496063104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761716.107680:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761716.107688:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761716/real 1789761716] req@ffff8dd203146300 x1876699496063104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761716.107696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496063104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761716.107701:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761716.107703:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761716.107705:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496063104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761716.107709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496063232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761716.107720:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761716.107725:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761716/real 1789761716] req@ffff8dd203147800 x1876699496063232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761716.107738:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496063232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761716.107743:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761716.107745:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761716.107747:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496063232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761716.107750:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496063360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761716.107759:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761716.107765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761716/real 1789761716] req@ffff8dd203145180 x1876699496063360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761716.107772:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496063360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761716.107777:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761716.107779:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761716.107781:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496063360:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761716.683302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761717.707353:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761718.731308:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761719.755345:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761720.779352:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761721.227317:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761721.227329:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761721.227334:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761721.227344:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1700 00000100:00080000:3.0:1789761721.227355:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761721.227362:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761721.227400:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496063488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761721.227429:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761721.227433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761721.227436:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761721.227440:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1700 00000100:00080000:3.0:1789761721.227451:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761721.227455:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761721.227469:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761721.227485:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496063616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761721.227487:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496063488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761721.227506:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761721.227510:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761721.227512:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761721.227517:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1700 00000100:00080000:3.0:1789761721.227521:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761721.227526:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761721.227532:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761721.227544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496063744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761721.227555:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761721.227558:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761721.227563:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761721.227564:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761721.227572:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1700 00000100:00080000:3.0:1789761721.227578:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761721.227582:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761721.227588:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761721.227601:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496063872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761721.227616:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761721/real 1789761721] req@ffff8dd203145c00 x1876699496063488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761721.227636:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496063488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761721.227647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761721.227651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761721.227656:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496063488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761721.227678:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496063616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761721.227703:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761721.227716:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761721/real 1789761721] req@ffff8dd203147480 x1876699496063616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761721.227728:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496063616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761721.227747:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761721.227750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761721.227753:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496063616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761721.227760:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496063744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761721.227778:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761721.227789:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761721/real 1789761721] req@ffff8dd203144000 x1876699496063744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761721.227801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496063744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761721.227817:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761721.227821:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761721.227824:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496063744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761721.227831:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496063872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761721.227851:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761721.227863:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761721/real 1789761721] req@ffff8dd203147100 x1876699496063872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761721.227874:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496063872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761721.227883:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761721.227886:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761721.227890:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496063872:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761721.803346:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761722.827365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761723.851337:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761724.875339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761725.899393:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761726.347291:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761726.347303:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761726.347307:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761726.347317:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1705 00000100:00080000:3.0:1789761726.347327:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761726.347333:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761726.347368:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496064000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761726.347393:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761726.347396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761726.347397:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761726.347400:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1705 00000100:00080000:3.0:1789761726.347403:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761726.347405:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761726.347413:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761726.347423:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496064128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761726.347430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761726.347433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761726.347434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761726.347437:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1705 00000100:00080000:3.0:1789761726.347439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761726.347444:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496064000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761726.347446:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761726.347454:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761726.347462:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496064256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761726.347470:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761726.347472:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761726.347474:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761726.347476:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1705 00000100:00080000:3.0:1789761726.347478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761726.347481:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761726.347484:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761726.347491:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496064384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761726.347513:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761726.347569:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761726/real 1789761726] req@ffff8dd203145880 x1876699496064000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761726.347589:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496064000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761726.347600:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761726.347604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761726.347608:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496064000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761726.347616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496064128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761726.347641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761726.347653:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761726/real 1789761726] req@ffff8dd203147100 x1876699496064128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761726.347662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496064128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761726.347668:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761726.347670:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761726.347672:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496064128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761726.347700:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496064256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761726.347723:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761726.347735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761726/real 1789761726] req@ffff8dd203144000 x1876699496064256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761726.347748:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496064256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761726.347754:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761726.347756:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761726.347758:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496064256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761726.347763:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496064384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761726.347777:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761726.347784:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761726/real 1789761726] req@ffff8dd203147480 x1876699496064384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761726.347792:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496064384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761726.347797:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761726.347800:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761726.347802:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496064384:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761726.923356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761727.947331:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761728.971335:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761729.995312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761731.019362:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761731.467335:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761731.467346:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761731.467351:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761731.467363:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1711 00000100:00080000:3.0:1789761731.467378:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761731.467387:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761731.467439:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496064512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761731.467472:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761731.467477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761731.467480:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761731.467485:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1711 00000100:00080000:3.0:1789761731.467491:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761731.467497:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761731.467511:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761731.467532:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496064512:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761731.467532:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496064640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761731.467562:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761731.467566:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761731.467570:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761731.467578:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1711 00000100:00080000:3.0:1789761731.467583:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761731.467589:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761731.467596:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761731.467611:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496064768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761731.467628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761731.467636:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761731.467641:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761731.467644:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761731.467652:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1711 00000100:00080000:3.0:1789761731.467658:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761731.467665:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761731.467673:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761731.467690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761731/real 1789761731] req@ffff8dd203145c00 x1876699496064512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761731.467693:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496064896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761731.467773:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496064512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761731.467784:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761731.467788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761731.467792:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496064512:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761731.467815:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496064640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761731.467836:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761731.467846:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761731/real 1789761731] req@ffff8dd203145180 x1876699496064640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761731.467856:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496064640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761731.467865:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761731.467869:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761731.467873:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496064640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761731.467879:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496064768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761731.467902:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761731.467915:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761731/real 1789761731] req@ffff8dd203147800 x1876699496064768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761731.467924:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496064768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761731.467930:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761731.467932:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761731.467934:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496064768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761731.467938:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496064896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761731.467952:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761731.467959:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761731/real 1789761731] req@ffff8dd203146300 x1876699496064896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761731.467966:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496064896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761731.467971:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761731.467973:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761731.467975:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496064896:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761732.043313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761733.067354:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761734.091318:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761735.115362:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761736.139316:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761736.587298:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761736.587310:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761736.587313:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761736.587324:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1716 00000100:00080000:0.0:1789761736.587334:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761736.587340:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761736.587376:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496065024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761736.587413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761736.587416:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761736.587417:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761736.587420:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1716 00000100:00080000:0.0:1789761736.587422:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761736.587425:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761736.587432:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761736.587441:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496065152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761736.587448:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761736.587451:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761736.587453:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761736.587456:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1716 00000100:00080000:0.0:1789761736.587458:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761736.587460:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761736.587463:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761736.587474:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496065280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761736.587475:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496065024:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761736.587492:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761736.587495:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761736.587497:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761736.587502:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1716 00000100:00080000:0.0:1789761736.587506:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761736.587510:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761736.587515:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761736.587527:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496065408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761736.587540:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761736.587576:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761736/real 1789761736] req@ffff8dd211837800 x1876699496065024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761736.587589:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496065024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761736.587597:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761736.587600:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761736.587603:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496065024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761736.587610:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496065152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761736.587624:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761736.587632:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761736/real 1789761736] req@ffff8dd211836300 x1876699496065152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761736.587640:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496065152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761736.587645:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761736.587647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761736.587649:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496065152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761736.587668:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496065280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789761736.587679:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761736.587685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761736/real 1789761736] req@ffff8dd211837100 x1876699496065280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761736.587692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496065280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761736.587698:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761736.587699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761736.587701:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496065280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761736.587704:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496065408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789761736.587714:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761736.587720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761736/real 1789761736] req@ffff8dd211837b80 x1876699496065408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761736.587728:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496065408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761736.587733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761736.587734:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761736.587736:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496065408:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761737.163339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761738.187295:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761739.211319:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761740.235344:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761741.259314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761741.707324:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761741.707337:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761741.707342:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761741.707354:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1721 00000100:00080000:1.0:1789761741.707368:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761741.707378:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761741.707431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496065536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761741.707458:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761741.707461:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761741.707462:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761741.707466:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1721 00000100:00080000:1.0:1789761741.707468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761741.707471:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761741.707480:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761741.707490:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496065664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761741.707503:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761741.707507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761741.707508:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761741.707511:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1721 00000100:00080000:1.0:1789761741.707515:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761741.707517:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496065536:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761741.707519:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761741.707525:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761741.707533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496065792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761741.707539:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761741.707542:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761741.707543:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761741.707546:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1721 00000100:00080000:1.0:1789761741.707549:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761741.707552:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761741.707554:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761741.707562:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496065920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761741.707603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761741.707659:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761741/real 1789761741] req@ffff8dd20504c380 x1876699496065536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761741.707682:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496065536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761741.707695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761741.707700:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761741.707705:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496065536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761741.707734:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496065664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761741.707766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761741.707781:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761741/real 1789761741] req@ffff8dd20504d880 x1876699496065664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761741.707796:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496065664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761741.707805:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761741.707809:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761741.707811:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496065664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761741.707816:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496065792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761741.707836:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761741.707849:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761741/real 1789761741] req@ffff8dd20504ed80 x1876699496065792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761741.707862:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496065792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761741.707871:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761741.707875:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761741.707878:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496065792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761741.707895:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496065920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761741.707920:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761741.707930:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761741/real 1789761741] req@ffff8dd20504f100 x1876699496065920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761741.707944:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496065920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761741.707950:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761741.707952:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761741.707954:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496065920:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761742.283292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761743.307334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761744.331350:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761745.355328:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761746.379339:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761746.827302:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761746.827311:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761746.827315:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761746.827324:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1726 00000100:00080000:3.0:1789761746.827333:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761746.827339:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761746.827373:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496066048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761746.827400:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761746.827403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761746.827404:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761746.827407:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1726 00000100:00080000:3.0:1789761746.827410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761746.827414:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761746.827425:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761746.827444:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496066048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761746.827444:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496066176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761746.827462:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761746.827466:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761746.827467:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761746.827470:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1726 00000100:00080000:3.0:1789761746.827474:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761746.827478:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761746.827483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761746.827493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496066304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761746.827500:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761746.827502:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761746.827504:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761746.827506:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1726 00000100:00080000:3.0:1789761746.827509:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761746.827511:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761746.827515:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761746.827523:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496066432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761746.827541:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496066176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761746.827569:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496066304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761746.827593:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496066432:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761747.403318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761748.427374:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761749.451368:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761749.899430:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789761749.899469:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761749.899483:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761749.899514:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761749.899521:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789761749.899527:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789761749.899611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761746/real 1789761749] req@ffff8dd203144380 x1876699496066176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761749.899651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496066176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761749.899665:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761749.899671:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761749.899675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496066176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761749.899685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761746/real 1789761749] req@ffff8dd203144a80 x1876699496066304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761749.899699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496066304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761749.899708:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761749.899712:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761749.899716:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496066304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761749.899723:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761746/real 1789761749] req@ffff8dd203144700 x1876699496066432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761749.899735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496066432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761749.899743:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761749.899747:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761749.899751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496066432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761749.899758:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761746/real 1789761749] req@ffff8dd203145f80 x1876699496066048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761749.899771:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496066048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761749.899778:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761749.899782:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761749.899785:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496066048:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789761750.475361:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761751.499356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761751.947229:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761751.947241:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761751.947244:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761751.947254:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1731 00000100:00080000:3.0:1789761751.947264:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761751.947271:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761751.947308:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496066560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761751.947342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761751.947345:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761751.947347:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761751.947351:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1731 00000100:00080000:3.0:1789761751.947354:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761751.947357:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761751.947364:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761751.947376:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496066688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761751.947387:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761751.947391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761751.947393:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761751.947399:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1731 00000100:00100000:0.0:1789761751.947403:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496066560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761751.947404:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761751.947413:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761751.947425:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761751.947442:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496066816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761751.947455:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761751.947459:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761751.947461:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761751.947469:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761751.947469:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1731 00000100:00080000:3.0:1789761751.947478:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761751.947482:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761751.947487:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761751.947499:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496066944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761751.947518:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761751/real 1789761751] req@ffff8dd203145500 x1876699496066560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761751.947538:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496066560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761751.947548:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761751.947552:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761751.947557:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496066560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761751.947565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496066688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761751.947586:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761751.947600:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761751/real 1789761751] req@ffff8dd203146a00 x1876699496066688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761751.947612:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496066688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761751.947621:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761751.947625:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761751.947629:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496066688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761751.947655:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496066816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761751.947678:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761751.947689:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761751/real 1789761751] req@ffff8dd203144e00 x1876699496066816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761751.947707:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496066816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761751.947716:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761751.947719:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761751.947723:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496066816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761751.947730:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496066944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761751.947750:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761751.947761:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761751/real 1789761751] req@ffff8dd203146d80 x1876699496066944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761751.947772:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496066944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761751.947780:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761751.947783:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761751.947786:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496066944:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761752.523335:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761753.547293:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761754.571331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761755.595313:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761756.619334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761757.067218:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761757.067227:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761757.067230:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761757.067241:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1736 00000100:00080000:3.0:1789761757.067252:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761757.067259:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761757.067298:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496067072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761757.067321:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761757.067323:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761757.067325:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761757.067328:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1736 00000100:00080000:3.0:1789761757.067331:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761757.067333:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761757.067341:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761757.067353:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496067200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761757.067360:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761757.067363:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761757.067364:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761757.067366:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1736 00000100:00080000:3.0:1789761757.067369:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761757.067373:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496067072:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761757.067374:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761757.067381:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761757.067388:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496067328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761757.067396:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761757.067398:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761757.067399:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761757.067401:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1736 00000100:00080000:3.0:1789761757.067403:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761757.067406:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761757.067409:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761757.067415:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496067456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761757.067443:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761757.067473:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761757/real 1789761757] req@ffff8dd203147b80 x1876699496067072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761757.067491:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496067072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761757.067498:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761757.067500:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761757.067503:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496067072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761757.067521:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496067200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761757.067536:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761757.067544:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761757/real 1789761757] req@ffff8dd203146d80 x1876699496067200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761757.067552:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496067200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761757.067558:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761757.067560:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761757.067562:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496067200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761757.067565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496067328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761757.067576:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761757.067582:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761757/real 1789761757] req@ffff8dd203144e00 x1876699496067328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761757.067589:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496067328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761757.067598:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761757.067600:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761757.067603:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496067328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761757.067606:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496067456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761757.067618:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761757.067628:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761757/real 1789761757] req@ffff8dd203146a00 x1876699496067456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761757.067640:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496067456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761757.067649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761757.067652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761757.067656:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496067456:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761757.643304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761758.667320:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761759.691327:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761760.715315:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761761.739406:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761762.187295:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761762.187306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761762.187311:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761762.187320:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1741 00000100:00080000:3.0:1789761762.187331:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761762.187340:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761762.187378:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496067584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761762.187406:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761762.187409:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761762.187412:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761762.187417:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1741 00000100:00080000:3.0:1789761762.187420:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761762.187424:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761762.187433:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761762.187447:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496067712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761762.187457:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496067584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761762.187463:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761762.187468:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761762.187471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761762.187476:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1741 00000100:00080000:3.0:1789761762.187480:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761762.187485:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761762.187491:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761762.187504:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496067840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761762.187515:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761762.187519:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761762.187521:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761762.187525:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761762.187531:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1741 00000100:00080000:3.0:1789761762.187536:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761762.187540:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761762.187545:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761762.187558:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496067968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761762.187562:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761762/real 1789761762] req@ffff8dd203145500 x1876699496067584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761762.187595:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496067584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761762.187607:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761762.187613:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761762.187617:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496067584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761762.187640:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496067712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761762.187658:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761762.187667:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761762/real 1789761762] req@ffff8dd203145f80 x1876699496067712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761762.187676:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496067712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761762.187681:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761762.187684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761762.187686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496067712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761762.187689:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496067840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761762.187700:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761762.187706:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761762/real 1789761762] req@ffff8dd203144700 x1876699496067840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761762.187714:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496067840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761762.187719:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761762.187721:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761762.187723:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496067840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761762.187726:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496067968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761762.187736:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761762.187741:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761762/real 1789761762] req@ffff8dd203144a80 x1876699496067968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761762.187748:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496067968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761762.187753:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761762.187755:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761762.187757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496067968:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761762.763339:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761763.787335:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761764.811313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761765.835342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761766.859359:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761767.307254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761767.307266:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761767.307271:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761767.307282:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1746 00000100:00080000:3.0:1789761767.307294:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761767.307302:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761767.307344:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496068096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761767.307370:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761767.307374:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761767.307376:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761767.307381:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1746 00000100:00080000:3.0:1789761767.307385:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761767.307389:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761767.307399:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761767.307415:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496068224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761767.307427:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496068096:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761767.307429:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761767.307438:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761767.307440:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761767.307445:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1746 00000100:00080000:3.0:1789761767.307449:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761767.307454:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761767.307460:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761767.307475:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496068352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761767.307486:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761767.307500:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789761767.307502:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761767.307503:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761767.307509:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1746 00000100:00080000:3.0:1789761767.307513:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761767.307517:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761767.307522:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761767.307532:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496068480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761767.307565:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761767/real 1789761767] req@ffff8dd203144380 x1876699496068096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761767.307586:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496068096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761767.307598:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761767.307603:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761767.307607:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496068096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761767.307631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496068224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761767.307659:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761767.307674:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761767/real 1789761767] req@ffff8dd203146300 x1876699496068224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761767.307692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496068224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761767.307706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761767.307709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761767.307714:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496068224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761767.307720:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496068352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761767.307741:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761767.307750:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761767/real 1789761767] req@ffff8dd203147800 x1876699496068352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761767.307760:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496068352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761767.307769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761767.307772:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761767.307776:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496068352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761767.307783:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496068480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761767.307803:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761767.307815:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761767/real 1789761767] req@ffff8dd203145180 x1876699496068480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761767.307828:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496068480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761767.307843:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761767.307847:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761767.307851:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496068480:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761767.883344:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761768.907291:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761769.931338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761770.955312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761771.979370:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761772.427257:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761772.427269:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761772.427274:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761772.427287:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1752 00000100:00080000:2.0:1789761772.427300:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761772.427310:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761772.427354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496068608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761772.427383:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761772.427387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761772.427389:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761772.427394:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1752 00000100:00080000:2.0:1789761772.427397:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761772.427401:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761772.427412:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761772.427430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496068736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761772.427441:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761772.427445:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761772.427446:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761772.427461:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1752 00000100:00080000:2.0:1789761772.427465:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761772.427473:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496068608:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761772.427474:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761772.427486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761772.427504:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496068864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761772.427520:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761772.427524:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761772.427526:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761772.427532:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1752 00000100:00080000:2.0:1789761772.427536:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761772.427539:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761772.427544:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761772.427554:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496068992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761772.427570:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761772.427620:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761772/real 1789761772] req@ffff8dd205265f80 x1876699496068608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761772.427640:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496068608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761772.427651:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761772.427655:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761772.427660:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496068608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761772.427669:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496068736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761772.427693:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761772.427706:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761772/real 1789761772] req@ffff8dd205266d80 x1876699496068736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761772.427718:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496068736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761772.427727:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761772.427731:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761772.427735:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496068736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761772.427760:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496068864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761772.427781:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761772.427792:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761772/real 1789761772] req@ffff8dd205265180 x1876699496068864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761772.427804:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496068864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761772.427813:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761772.427816:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761772.427820:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496068864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761772.427826:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496068992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761772.427845:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761772.427856:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761772/real 1789761772] req@ffff8dd205266300 x1876699496068992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761772.427867:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496068992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761772.427876:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761772.427879:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761772.427882:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496068992:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761773.003351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761774.027374:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761775.051356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761776.075375:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761777.099308:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761777.547310:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761777.547323:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761777.547327:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761777.547338:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1757 00000100:00080000:2.0:1789761777.547350:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761777.547357:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761777.547395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496069120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761777.547425:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761777.547429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761777.547431:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761777.547435:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1757 00000100:00080000:2.0:1789761777.547439:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761777.547442:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761777.547460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761777.547474:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496069248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761777.547481:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496069120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761777.547489:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761777.547494:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761777.547496:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761777.547501:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1757 00000100:00080000:2.0:1789761777.547505:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761777.547509:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761777.547515:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761777.547529:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496069376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761777.547540:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761777.547543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761777.547545:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761777.547550:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1757 00000100:00080000:2.0:1789761777.547554:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761777.547558:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761777.547564:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789761777.547565:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761777.547581:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496069504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761777.547614:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761777/real 1789761777] req@ffff8dd205264700 x1876699496069120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761777.547631:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496069120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761777.547640:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761777.547643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761777.547646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496069120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761777.547666:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496069248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761777.547691:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761777.547706:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761777/real 1789761777] req@ffff8dd205266a00 x1876699496069248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761777.547720:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496069248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761777.547730:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761777.547732:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761777.547735:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496069248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761777.547740:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496069376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761777.547757:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761777.547765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761777/real 1789761777] req@ffff8dd205265880 x1876699496069376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761777.547794:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496069376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761777.547803:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761777.547806:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761777.547810:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496069376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761777.547816:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496069504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761777.547838:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761777.547848:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761777/real 1789761777] req@ffff8dd205264000 x1876699496069504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761777.547857:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496069504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761777.547879:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761777.547883:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761777.547886:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496069504:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761778.123347:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761779.147359:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761780.171338:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761781.195355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761782.219358:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761782.667294:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761782.667306:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761782.667309:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761782.667320:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1762 00000100:00080000:3.0:1789761782.667330:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761782.667338:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761782.667372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496069632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761782.667397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761782.667401:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761782.667403:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761782.667406:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1762 00000100:00080000:3.0:1789761782.667409:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761782.667412:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761782.667422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761782.667433:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496069760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761782.667444:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496069632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761782.667445:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761782.667453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761782.667455:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761782.667459:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1762 00000100:00080000:3.0:1789761782.667463:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761782.667467:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761782.667472:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761782.667482:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496069888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761782.667491:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761782.667494:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761782.667496:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761782.667499:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1762 00000100:00080000:3.0:1789761782.667502:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761782.667508:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761782.667509:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1789761782.667513:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761782.667525:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496070016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761782.667553:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761782/real 1789761782] req@ffff8dd203145c00 x1876699496069632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761782.667570:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496069632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761782.667580:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761782.667584:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761782.667588:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496069632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761782.667608:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496069760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761782.667628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761782.667639:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761782/real 1789761782] req@ffff8dd203147480 x1876699496069760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761782.667650:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496069760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761782.667659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761782.667662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761782.667666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496069760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761782.667670:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496069888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761782.667690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761782.667700:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761782/real 1789761782] req@ffff8dd203144000 x1876699496069888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761782.667711:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496069888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761782.667718:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761782.667720:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761782.667723:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496069888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761782.667729:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496070016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761782.667743:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761782.667751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761782/real 1789761782] req@ffff8dd203147100 x1876699496070016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761782.667761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496070016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761782.667769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761782.667772:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761782.667774:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496070016:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761783.243333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761784.267320:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761785.291301:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761786.315289:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761787.339302:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761787.787259:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761787.787272:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761787.787276:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761787.787284:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1767 00000100:00080000:2.0:1789761787.787293:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761787.787298:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761787.787327:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496070144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761787.787346:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761787.787348:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761787.787349:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761787.787352:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1767 00000100:00080000:2.0:1789761787.787354:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761787.787356:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761787.787363:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761787.787372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496070272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761787.787377:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496070144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761787.787382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761787.787385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761787.787387:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761787.787392:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1767 00000100:00080000:2.0:1789761787.787395:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761787.787399:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761787.787404:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761787.787417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496070400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761787.787427:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761787.787430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761787.787431:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761787.787436:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1767 00000800:00000100:0.0:1789761787.787439:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761787.787440:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761787.787444:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761787.787448:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761787.787456:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496070528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761787.787483:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761787/real 1789761787] req@ffff8dd205267b80 x1876699496070144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761787.787502:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496070144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761787.787509:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761787.787512:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761787.787514:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496070144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761787.787532:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496070272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761787.787549:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761787.787560:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761787/real 1789761787] req@ffff8dd205264e00 x1876699496070272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761787.787579:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496070272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761787.787587:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761787.787590:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761787.787593:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496070272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761787.787599:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496070400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761787.787619:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761787.787630:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761787/real 1789761787] req@ffff8dd205264380 x1876699496070400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761787.787642:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496070400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761787.787649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761787.787652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761787.787655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496070400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761787.787661:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496070528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761787.787680:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761787.787689:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761787/real 1789761787] req@ffff8dd205267800 x1876699496070528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761787.787697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496070528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761787.787703:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761787.787704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761787.787706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496070528:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761788.363308:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761789.387309:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761790.411296:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761791.435347:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761792.459294:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761792.907283:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761792.907294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761792.907299:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761792.907311:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1772 00000100:00080000:1.0:1789761792.907332:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761792.907340:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761792.907381:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496070656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761792.907408:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761792.907412:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761792.907414:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761792.907418:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1772 00000100:00080000:1.0:1789761792.907422:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761792.907426:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761792.907437:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761792.907454:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496070656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761792.907455:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496070784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761792.907471:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761792.907475:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761792.907476:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761792.907480:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1772 00000100:00080000:1.0:1789761792.907484:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761792.907489:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761792.907494:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761792.907506:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496070912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761792.907517:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761792.907521:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761792.907525:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761792.907526:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761792.907537:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1772 00000100:00080000:1.0:1789761792.907542:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761792.907548:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761792.907552:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761792.907563:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496071040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761792.907587:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761792/real 1789761792] req@ffff8dd20504ce00 x1876699496070656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761792.907603:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496070656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761792.907611:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761792.907614:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761792.907617:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496070656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761792.907636:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496070784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761792.907656:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761792.907670:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761792/real 1789761792] req@ffff8dd20504f800 x1876699496070784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761792.907683:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496070784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761792.907692:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761792.907697:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761792.907701:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496070784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761792.907707:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496070912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761792.907726:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761792.907734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761792/real 1789761792] req@ffff8dd20504fb80 x1876699496070912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761792.907758:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496070912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761792.907768:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761792.907771:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761792.907775:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496070912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761792.907781:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496071040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761792.907799:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761792.907807:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761792/real 1789761792] req@ffff8dd20504df80 x1876699496071040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761792.907815:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496071040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761792.907820:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761792.907823:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761792.907825:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496071040:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761793.483365:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761794.507308:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761795.531316:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761796.555336:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761797.579372:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761798.027316:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761798.027327:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761798.027331:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761798.027347:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1777 00000100:00080000:3.0:1789761798.027356:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761798.027362:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761798.027396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496071168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761798.027418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761798.027421:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761798.027422:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761798.027425:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1777 00000100:00080000:3.0:1789761798.027428:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761798.027430:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761798.027442:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761798.027455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496071296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761798.027460:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496071168:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761798.027469:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761798.027472:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761798.027474:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761798.027477:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1777 00000100:00080000:3.0:1789761798.027480:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761798.027483:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761798.027487:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761798.027498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496071424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761798.027506:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761798.027508:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761798.027509:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761798.027511:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1777 00000100:00080000:3.0:1789761798.027514:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761798.027517:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761798.027519:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761798.027525:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761798.027529:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496071552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761798.027568:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761798/real 1789761798] req@ffff8dd203145880 x1876699496071168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761798.027583:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496071168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761798.027591:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761798.027594:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761798.027597:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496071168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761798.027615:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496071296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761798.027630:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761798.027638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761798/real 1789761798] req@ffff8dd203147100 x1876699496071296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761798.027646:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496071296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761798.027651:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761798.027654:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761798.027656:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496071296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761798.027659:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496071424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761798.027670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761798.027676:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761798/real 1789761798] req@ffff8dd203144000 x1876699496071424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761798.027688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496071424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761798.027693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761798.027695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761798.027697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496071424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761798.027700:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496071552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761798.027727:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761798.027735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761798/real 1789761798] req@ffff8dd203147480 x1876699496071552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761798.027743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496071552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761798.027748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761798.027750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761798.027753:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496071552:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761798.603339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761799.627320:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761800.651359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761801.675353:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761802.699404:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761803.147334:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761803.147345:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761803.147350:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761803.147364:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1782 00000100:00080000:2.0:1789761803.147376:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761803.147382:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761803.147424:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496071680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761803.147451:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761803.147454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761803.147456:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761803.147459:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1782 00000100:00080000:2.0:1789761803.147461:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761803.147464:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761803.147473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761803.147485:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496071808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761803.147493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761803.147495:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761803.147498:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761803.147501:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1782 00000100:00080000:2.0:1789761803.147505:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761803.147513:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761803.147515:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496071680:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789761803.147519:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761803.147539:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496071936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761803.147548:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761803.147551:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761803.147553:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761803.147556:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1782 00000100:00080000:2.0:1789761803.147559:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761803.147563:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761803.147568:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761803.147576:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496072064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761803.147599:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761803.147637:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761803/real 1789761803] req@ffff8dd205267100 x1876699496071680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761803.147651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496071680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761803.147660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761803.147663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761803.147677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496071680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761803.147685:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496071808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761803.147703:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761803.147718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761803/real 1789761803] req@ffff8dd205264a80 x1876699496071808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761803.147727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496071808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761803.147733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761803.147735:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761803.147737:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496071808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761803.147758:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496071936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761803.147771:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761803.147777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761803/real 1789761803] req@ffff8dd205267480 x1876699496071936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761803.147785:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496071936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761803.147790:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761803.147793:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761803.147796:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496071936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761803.147803:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496072064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761803.147822:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761803.147834:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761803/real 1789761803] req@ffff8dd205266680 x1876699496072064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761803.147847:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496072064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761803.147856:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761803.147859:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761803.147863:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496072064:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761803.723380:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761804.747364:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761805.771320:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761806.795299:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761807.819288:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761808.267236:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761808.267245:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761808.267249:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761808.267258:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1787 00000100:00080000:2.0:1789761808.267268:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761808.267274:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761808.267310:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496072192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761808.267336:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761808.267339:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761808.267340:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761808.267343:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1787 00000100:00080000:2.0:1789761808.267345:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761808.267348:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761808.267356:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761808.267365:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496072320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761808.267373:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761808.267375:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761808.267377:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761808.267379:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1787 00000100:00080000:2.0:1789761808.267381:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761808.267385:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496072192:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761808.267386:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761808.267393:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761808.267400:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496072448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761808.267407:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761808.267410:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761808.267411:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761808.267414:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1787 00000100:00080000:2.0:1789761808.267417:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761808.267419:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761808.267422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761808.267429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496072576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761808.267447:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761808.267484:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761808/real 1789761808] req@ffff8dd205265500 x1876699496072192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761808.267497:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496072192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761808.267505:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761808.267507:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761808.267510:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496072192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761808.267515:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496072320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761808.267529:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761808.267537:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761808/real 1789761808] req@ffff8dd205266680 x1876699496072320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761808.267545:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496072320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761808.267550:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761808.267552:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761808.267554:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496072320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761808.267582:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496072448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761808.267596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761808.267602:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761808/real 1789761808] req@ffff8dd205267480 x1876699496072448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761808.267610:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496072448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761808.267615:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761808.267617:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761808.267619:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496072448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761808.267623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496072576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761808.267650:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761808.267657:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761808/real 1789761808] req@ffff8dd205264a80 x1876699496072576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761808.267672:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496072576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761808.267680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761808.267683:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761808.267687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496072576:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761808.843348:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761809.867321:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761810.891352:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761811.915325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761812.939384:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761813.387318:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761813.387331:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761813.387335:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761813.387348:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1792 00000100:00080000:2.0:1789761813.387364:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761813.387371:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761813.387404:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496072704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761813.387434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761813.387438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761813.387440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761813.387445:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1792 00000100:00080000:2.0:1789761813.387449:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761813.387454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761813.387464:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761813.387476:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496072832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761813.387485:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761813.387488:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761813.387489:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761813.387492:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1792 00000100:00080000:2.0:1789761813.387494:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761813.387497:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761813.387501:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761813.387510:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496072960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761813.387518:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496072704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761813.387522:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761813.387536:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761813.387538:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761813.387542:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1792 00000100:00080000:2.0:1789761813.387544:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761813.387546:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761813.387549:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761813.387559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496073088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761813.387664:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496072832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761813.387704:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496072960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761813.387723:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496073088:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761813.963349:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761814.987322:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761816.011328:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761816.459465:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789761816.459508:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761816.459525:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761816.459579:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761816.459591:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789761816.459602:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789761816.459633:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761813/real 1789761816] req@ffff8dd205264380 x1876699496072960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761816.459661:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496072960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761816.459682:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761816.459687:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761816.459692:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496072960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789761816.459708:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761813/real 1789761816] req@ffff8dd205264e00 x1876699496073088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761816.459724:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496073088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761816.459733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761816.459737:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761816.459742:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496073088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789761816.459750:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761813/real 1789761816] req@ffff8dd205267100 x1876699496072704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761816.459763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496072704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761816.459772:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761816.459776:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761816.459780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496072704:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761816.459787:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761813/real 1789761816] req@ffff8dd205267800 x1876699496072832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761816.459799:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496072832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761816.459808:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761816.459812:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789761816.459816:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496072832:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761817.035395:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761818.059320:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761818.507253:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761818.507266:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761818.507271:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761818.507282:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1798 00000100:00080000:1.0:1789761818.507291:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761818.507297:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761818.507326:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496073216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761818.507350:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761818.507354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761818.507355:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761818.507358:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1798 00000100:00080000:1.0:1789761818.507360:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761818.507363:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761818.507372:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761818.507383:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496073344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761818.507388:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496073216:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761818.507403:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761818.507407:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761818.507420:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761818.507429:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1798 00000100:00080000:1.0:1789761818.507448:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761818.507453:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761818.507459:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789761818.507461:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761818.507492:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496073472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761818.507504:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761818/real 1789761818] req@ffff8dd20504d500 x1876699496073216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761818.507507:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761818.507512:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761818.507514:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761818.507523:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1798 00000100:00080000:1.0:1789761818.507531:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761818.507536:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761818.507542:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496073216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761818.507549:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761818.507574:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789761818.507577:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496073600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761818.507582:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761818.507592:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496073216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761818.507623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496073344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761818.507649:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761818.507664:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761818/real 1789761818] req@ffff8dd20504e300 x1876699496073344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761818.507678:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496073344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761818.507688:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761818.507692:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761818.507696:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496073344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761818.507703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496073472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761818.507725:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761818.507737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761818/real 1789761818] req@ffff8dd20504c700 x1876699496073472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761818.507749:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496073472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761818.507758:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761818.507762:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761818.507765:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496073472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761818.507771:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496073600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761818.507791:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761818.507802:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761818/real 1789761818] req@ffff8dd20504e680 x1876699496073600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761818.507813:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496073600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761818.507821:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761818.507825:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761818.507828:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496073600:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761819.083353:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761820.107445:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761821.131313:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761822.155347:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761823.179346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761823.627269:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761823.627281:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761823.627286:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761823.627298:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1803 00000100:00080000:1.0:1789761823.627313:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761823.627321:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761823.627368:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496073728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761823.627409:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761823.627413:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761823.627415:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761823.627420:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1803 00000100:00080000:1.0:1789761823.627426:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761823.627431:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761823.627445:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761823.627453:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496073728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761823.627465:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496073856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761823.627478:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761823.627482:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761823.627484:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761823.627489:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1803 00000100:00080000:1.0:1789761823.627493:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761823.627498:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761823.627504:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761823.627533:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496073984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761823.627545:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761823.627548:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761823.627550:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761823.627557:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1803 00000800:00000100:2.0:1789761823.627560:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761823.627564:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761823.627570:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761823.627577:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761823.627591:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496074112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761823.627614:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761823/real 1789761823] req@ffff8dd20504d180 x1876699496073728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761823.627637:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496073728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761823.627650:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761823.627655:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761823.627659:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496073728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761823.627690:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496073856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761823.627717:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761823.627734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761823/real 1789761823] req@ffff8dd20504e680 x1876699496073856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761823.627748:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496073856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761823.627759:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761823.627763:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761823.627767:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496073856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789761823.627774:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496073984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789761823.627799:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761823.627813:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761823/real 1789761823] req@ffff8dd20504c700 x1876699496073984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761823.627829:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496073984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761823.627850:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761823.627854:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761823.627858:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496073984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789761823.627865:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496074112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761823.627888:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761823.627912:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761823/real 1789761823] req@ffff8dd20504e300 x1876699496074112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761823.627924:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496074112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761823.627933:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761823.627936:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761823.627940:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496074112:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761824.203349:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761825.227352:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761826.251453:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761827.275297:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761828.299309:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761828.747268:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761828.747278:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761828.747283:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761828.747294:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1808 00000100:00080000:1.0:1789761828.747307:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761828.747314:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761828.747356:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496074240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761828.747383:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761828.747386:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761828.747388:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761828.747391:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1808 00000100:00080000:1.0:1789761828.747398:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761828.747401:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761828.747410:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761828.747422:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496074368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761828.747435:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761828.747439:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761828.747440:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761828.747443:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1808 00000100:00080000:1.0:1789761828.747446:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761828.747448:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761828.747452:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761828.747463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496074496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761828.747465:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496074240:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761828.747497:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761828.747500:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761828.747503:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761828.747509:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1808 00000100:00080000:1.0:1789761828.747512:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761828.747516:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761828.747520:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761828.747532:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496074624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761828.747562:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761828.747607:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761828/real 1789761828] req@ffff8dd20504d500 x1876699496074240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761828.747621:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496074240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761828.747630:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761828.747634:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761828.747637:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496074240:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761828.747643:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496074368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761828.747659:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761828.747667:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761828/real 1789761828] req@ffff8dd20504df80 x1876699496074368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761828.747675:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496074368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761828.747681:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761828.747684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761828.747686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496074368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789761828.747706:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496074496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789761828.747719:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761828.747725:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761828/real 1789761828] req@ffff8dd20504fb80 x1876699496074496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761828.747732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496074496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761828.747738:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761828.747740:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761828.747742:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496074496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789761828.747746:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496074624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761828.747757:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761828.747763:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761828/real 1789761828] req@ffff8dd20504f800 x1876699496074624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761828.747770:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496074624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761828.747775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761828.747777:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761828.747779:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496074624:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761829.323343:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761830.347313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761831.371297:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761832.395357:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761833.419331:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761833.867259:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761833.867274:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761833.867278:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761833.867288:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1813 00000100:00080000:1.0:1789761833.867299:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761833.867307:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761833.867348:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496074752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761833.867379:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761833.867383:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761833.867385:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761833.867389:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1813 00000100:00080000:1.0:1789761833.867393:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761833.867398:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761833.867407:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761833.867425:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496074880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761833.867432:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496074752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761833.867456:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761833.867459:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761833.867461:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761833.867465:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1813 00000100:00080000:1.0:1789761833.867469:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761833.867472:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761833.867477:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761833.867487:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496075008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761833.867495:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761833.867497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761833.867499:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761833.867502:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1813 00000100:00080000:1.0:1789761833.867505:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761833.867510:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1789761833.867512:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789761833.867515:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761833.867524:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496075136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761833.867559:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761833/real 1789761833] req@ffff8dd20504ce00 x1876699496074752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761833.867577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496074752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761833.867588:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761833.867592:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761833.867596:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496074752:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761833.867618:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496074880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761833.867643:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761833.867656:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761833/real 1789761833] req@ffff8dd20504f100 x1876699496074880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761833.867669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496074880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761833.867686:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761833.867690:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761833.867694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496074880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789761833.867703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496075008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789761833.867722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761833.867734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761833/real 1789761833] req@ffff8dd20504ed80 x1876699496075008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761833.867746:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496075008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761833.867753:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761833.867757:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761833.867760:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496075008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789761833.867765:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496075136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761833.867783:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761833.867793:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761833/real 1789761833] req@ffff8dd20504d880 x1876699496075136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761833.867812:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496075136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761833.867820:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761833.867823:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761833.867826:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496075136:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761834.443343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761835.467291:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761836.491352:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761837.515430:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761838.539465:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761838.987282:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761838.987297:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761838.987301:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761838.987313:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1818 00000100:00080000:2.0:1789761838.987326:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761838.987332:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761838.987365:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496075264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761838.987391:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761838.987394:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761838.987396:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761838.987411:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1818 00000100:00080000:2.0:1789761838.987415:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761838.987418:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761838.987427:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761838.987443:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496075264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761838.987443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496075392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761838.987458:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761838.987461:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761838.987462:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761838.987466:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1818 00000100:00080000:2.0:1789761838.987468:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761838.987470:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761838.987474:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761838.987483:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496075520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761838.987490:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761838.987493:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761838.987494:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761838.987497:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1818 00000100:00080000:2.0:1789761838.987499:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761838.987501:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761838.987504:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761838.987511:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761838.987512:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496075648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761838.987561:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761838/real 1789761838] req@ffff8dd205267b80 x1876699496075264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761838.987582:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496075264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761838.987594:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761838.987598:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761838.987603:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496075264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761838.987627:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496075392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761838.987650:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761838.987661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761838/real 1789761838] req@ffff8dd205264000 x1876699496075392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761838.987670:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496075392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761838.987676:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761838.987679:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761838.987681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496075392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761838.987687:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496075520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761838.987709:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761838.987722:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761838/real 1789761838] req@ffff8dd205265880 x1876699496075520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761838.987735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496075520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761838.987744:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761838.987747:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761838.987751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496075520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761838.987768:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496075648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761838.987792:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761838.987800:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761838/real 1789761838] req@ffff8dd205266a00 x1876699496075648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761838.987809:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496075648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761838.987814:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761838.987816:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761838.987818:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496075648:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761839.563368:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761840.587333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761841.611306:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761842.635330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761843.659320:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761844.107327:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761844.107336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761844.107340:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761844.107349:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1823 00000100:00080000:1.0:1789761844.107358:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761844.107376:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761844.107413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496075776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761844.107438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761844.107441:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761844.107443:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761844.107446:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1823 00000100:00080000:1.0:1789761844.107448:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761844.107451:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761844.107458:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761844.107468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496075904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761844.107475:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761844.107478:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761844.107480:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761844.107483:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1823 00000100:00080000:1.0:1789761844.107487:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761844.107488:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496075776:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761844.107492:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761844.107498:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761844.107507:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496076032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761844.107514:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761844.107517:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761844.107518:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761844.107520:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1823 00000100:00080000:1.0:1789761844.107523:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761844.107525:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761844.107529:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761844.107535:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496076160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761844.107549:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761844.107586:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761844/real 1789761844] req@ffff8dd20504c380 x1876699496075776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761844.107599:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496075776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761844.107607:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761844.107610:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761844.107612:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496075776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761844.107617:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496075904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761844.107641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761844.107652:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761844/real 1789761844] req@ffff8dd20504dc00 x1876699496075904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761844.107661:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496075904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761844.107666:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761844.107668:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761844.107670:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496075904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761844.107691:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496076032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761844.107711:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761844.107724:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761844/real 1789761844] req@ffff8dd20504c000 x1876699496076032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761844.107737:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496076032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761844.107745:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761844.107749:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761844.107752:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496076032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761844.107759:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496076160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761844.107780:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761844.107792:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761844/real 1789761844] req@ffff8dd20504ea00 x1876699496076160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761844.107802:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496076160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761844.107812:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761844.107814:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761844.107816:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496076160:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761844.683380:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761845.707360:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761846.731313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761847.755374:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761848.779354:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761849.227279:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761849.227288:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761849.227292:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761849.227301:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1828 00000100:00080000:1.0:1789761849.227311:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761849.227317:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761849.227367:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496076288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761849.227399:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761849.227403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761849.227405:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761849.227410:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1828 00000100:00080000:1.0:1789761849.227414:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761849.227418:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761849.227428:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761849.227446:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496076416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761849.227452:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496076288:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761849.227459:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761849.227463:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761849.227464:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761849.227468:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1828 00000100:00080000:1.0:1789761849.227471:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761849.227474:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761849.227478:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761849.227488:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496076544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761849.227495:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761849.227497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761849.227498:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761849.227501:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1828 00000100:00080000:1.0:1789761849.227504:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761849.227507:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761849.227511:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761849.227514:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761849.227523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496076672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761849.227554:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761849/real 1789761849] req@ffff8dd20504ca80 x1876699496076288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761849.227568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496076288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761849.227576:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761849.227579:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761849.227582:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496076288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761849.227600:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496076416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761849.227615:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761849.227623:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761849/real 1789761849] req@ffff8dd20504ea00 x1876699496076416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761849.227631:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496076416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761849.227641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761849.227643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761849.227645:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496076416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761849.227649:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496076544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761849.227661:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761849.227666:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761849/real 1789761849] req@ffff8dd20504c000 x1876699496076544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761849.227674:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496076544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761849.227679:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761849.227680:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761849.227682:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496076544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761849.227686:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496076672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761849.227695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761849.227701:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761849/real 1789761849] req@ffff8dd20504dc00 x1876699496076672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761849.227708:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496076672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761849.227713:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761849.227714:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761849.227717:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496076672:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761849.803360:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761850.827296:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761851.851374:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761852.875337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761853.899410:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761854.347254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761854.347265:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761854.347269:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761854.347279:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1833 00000100:00080000:1.0:1789761854.347302:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761854.347308:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761854.347354:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496076800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761854.347382:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761854.347385:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761854.347386:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761854.347389:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1833 00000100:00080000:1.0:1789761854.347392:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761854.347394:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761854.347401:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761854.347410:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496076928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761854.347418:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761854.347421:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761854.347422:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761854.347425:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1833 00000100:00080000:1.0:1789761854.347427:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761854.347430:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761854.347434:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761854.347439:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496076800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761854.347444:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496077056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761854.347453:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761854.347455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761854.347456:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761854.347460:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1833 00000100:00080000:1.0:1789761854.347462:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761854.347465:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761854.347467:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761854.347475:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496077184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761854.347502:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761854.347540:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761854/real 1789761854] req@ffff8dd20504c380 x1876699496076800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761854.347553:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496076800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761854.347561:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761854.347564:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761854.347567:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496076800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761854.347572:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496076928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761854.347585:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761854.347593:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761854/real 1789761854] req@ffff8dd20504d880 x1876699496076928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761854.347601:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496076928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761854.347606:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761854.347608:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761854.347610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496076928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761854.347643:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496077056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761854.347654:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761854.347661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761854/real 1789761854] req@ffff8dd20504ed80 x1876699496077056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761854.347668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496077056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761854.347673:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761854.347675:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761854.347677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496077056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761854.347680:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496077184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761854.347690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761854.347695:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761854/real 1789761854] req@ffff8dd20504f100 x1876699496077184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761854.347703:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496077184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761854.347708:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761854.347709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761854.347711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496077184:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761854.923466:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761855.947358:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761856.971310:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761857.995370:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761859.019388:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761859.467317:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761859.467326:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761859.467330:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761859.467340:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1839 00000100:00080000:1.0:1789761859.467350:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761859.467357:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761859.467399:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496077312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761859.467432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761859.467435:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761859.467437:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761859.467440:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1839 00000100:00080000:1.0:1789761859.467443:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761859.467447:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761859.467455:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761859.467466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496077440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761859.467475:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761859.467480:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496077312:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761859.467480:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761859.467482:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761859.467489:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1839 00000100:00080000:1.0:1789761859.467494:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761859.467496:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761859.467500:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761859.467508:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496077568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761859.467514:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761859.467516:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761859.467518:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761859.467520:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1839 00000100:00080000:1.0:1789761859.467523:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761859.467525:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761859.467528:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761859.467535:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496077696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761859.467555:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761859.467591:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761859/real 1789761859] req@ffff8dd20504ce00 x1876699496077312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761859.467604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496077312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761859.467612:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761859.467615:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761859.467618:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496077312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761859.467637:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496077440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761859.467652:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761859.467660:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761859/real 1789761859] req@ffff8dd20504f800 x1876699496077440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761859.467679:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496077440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761859.467684:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761859.467687:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761859.467689:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496077440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761859.467692:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496077568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761859.467703:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761859.467709:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761859/real 1789761859] req@ffff8dd20504fb80 x1876699496077568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761859.467721:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496077568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761859.467726:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761859.467728:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761859.467730:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496077568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761859.467734:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496077696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761859.467744:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761859.467750:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761859/real 1789761859] req@ffff8dd20504df80 x1876699496077696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761859.467757:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496077696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761859.467762:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761859.467764:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761859.467766:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496077696:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761860.043378:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761861.067396:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761862.091326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761863.115461:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761864.139357:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761864.587291:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761864.587299:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761864.587303:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761864.587313:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1844 00000100:00080000:1.0:1789761864.587341:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761864.587348:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761864.587381:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496077824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761864.587406:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761864.587410:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761864.587411:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761864.587414:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1844 00000100:00080000:1.0:1789761864.587417:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761864.587419:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761864.587427:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761864.587437:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496077952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761864.587447:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761864.587451:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496077824:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761864.587451:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761864.587455:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761864.587463:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1844 00000100:00080000:1.0:1789761864.587465:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761864.587467:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761864.587471:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761864.587480:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496078080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761864.587488:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761864.587491:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761864.587492:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761864.587495:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1844 00000100:00080000:1.0:1789761864.587514:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761864.587516:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761864.587520:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761864.587526:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761864.587530:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496078208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761864.587594:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761864/real 1789761864] req@ffff8dd20504d500 x1876699496077824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761864.587609:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496077824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761864.587617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761864.587620:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761864.587623:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496077824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761864.587658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496077952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761864.587676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761864.587692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761864/real 1789761864] req@ffff8dd20504e300 x1876699496077952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761864.587702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496077952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761864.587708:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761864.587710:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761864.587712:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496077952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761864.587716:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496078080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761864.587729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761864.587735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761864/real 1789761864] req@ffff8dd20504c700 x1876699496078080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761864.587743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496078080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761864.587749:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761864.587751:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761864.587753:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496078080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761864.587757:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496078208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761864.587767:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761864.587774:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761864/real 1789761864] req@ffff8dd20504e680 x1876699496078208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761864.587782:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496078208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761864.587787:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761864.587788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761864.587790:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496078208:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761865.163349:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761866.187435:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761867.211382:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761868.235293:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761869.259343:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761869.707301:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761869.707311:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761869.707315:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761869.707326:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1849 00000100:00080000:2.0:1789761869.707338:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761869.707344:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761869.707387:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496078336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761869.707413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761869.707417:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761869.707420:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761869.707428:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1849 00000100:00080000:2.0:1789761869.707432:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761869.707437:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761869.707448:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761869.707463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496078464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761869.707468:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496078336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761869.707475:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761869.707478:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761869.707480:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761869.707484:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1849 00000100:00080000:2.0:1789761869.707487:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761869.707502:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761869.707508:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761869.707518:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496078592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761869.707528:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761869.707531:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761869.707533:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761869.707536:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1849 00000100:00080000:2.0:1789761869.707539:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761869.707542:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761869.707543:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761869.707548:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761869.707556:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496078720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761869.707589:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761869/real 1789761869] req@ffff8dd205264700 x1876699496078336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761869.707610:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496078336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761869.707620:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761869.707623:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761869.707626:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496078336:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761869.707656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496078464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761869.707677:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761869.707687:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761869/real 1789761869] req@ffff8dd205266300 x1876699496078464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761869.707696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496078464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761869.707702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761869.707704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761869.707706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496078464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761869.707710:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496078592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761869.707722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761869.707728:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761869/real 1789761869] req@ffff8dd205265180 x1876699496078592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761869.707736:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496078592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761869.707741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761869.707743:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761869.707745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496078592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761869.707749:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496078720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761869.707759:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761869.707765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761869/real 1789761869] req@ffff8dd205266d80 x1876699496078720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761869.707778:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496078720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761869.707783:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761869.707785:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761869.707786:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496078720:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761870.283303:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761871.307326:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761872.331317:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761873.355331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761874.379359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761874.827262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761874.827272:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761874.827277:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761874.827286:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1854 00000100:00080000:2.0:1789761874.827311:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761874.827318:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761874.827351:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496078848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761874.827387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761874.827390:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761874.827392:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761874.827395:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1854 00000100:00080000:2.0:1789761874.827398:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761874.827401:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761874.827409:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761874.827418:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496078848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761874.827431:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496078976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761874.827441:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761874.827444:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761874.827445:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761874.827448:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1854 00000100:00080000:2.0:1789761874.827451:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761874.827454:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761874.827458:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761874.827467:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496079104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761874.827480:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761874.827484:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761874.827485:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761874.827489:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761874.827492:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1854 00000100:00080000:2.0:1789761874.827496:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761874.827500:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761874.827503:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761874.827512:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496079232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761874.827546:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761874/real 1789761874] req@ffff8dd205265f80 x1876699496078848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761874.827563:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496078848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761874.827572:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761874.827576:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761874.827579:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496078848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761874.827615:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496078976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761874.827631:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761874.827639:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761874/real 1789761874] req@ffff8dd205266d80 x1876699496078976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761874.827652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496078976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761874.827657:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761874.827659:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761874.827661:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496078976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761874.827665:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496079104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761874.827677:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761874.827683:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761874/real 1789761874] req@ffff8dd205265180 x1876699496079104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761874.827690:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496079104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761874.827695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761874.827697:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761874.827698:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496079104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761874.827702:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496079232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761874.827722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761874.827729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761874/real 1789761874] req@ffff8dd205266300 x1876699496079232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761874.827741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496079232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761874.827755:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761874.827759:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761874.827763:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496079232:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761875.403334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761876.427349:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761877.451380:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761878.475377:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761879.499337:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761879.947270:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761879.947280:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761879.947284:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761879.947293:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1859 00000100:00080000:1.0:1789761879.947303:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761879.947309:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761879.947343:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496079360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761879.947368:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761879.947371:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761879.947372:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761879.947375:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1859 00000100:00080000:1.0:1789761879.947377:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761879.947379:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761879.947387:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761879.947398:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496079488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761879.947416:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761879.947419:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761879.947420:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761879.947422:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1859 00000100:00080000:1.0:1789761879.947423:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761879.947426:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761879.947431:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496079360:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789761879.947432:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761879.947443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496079616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761879.947450:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761879.947453:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761879.947454:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761879.947456:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1859 00000100:00080000:1.0:1789761879.947460:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761879.947462:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761879.947465:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761879.947473:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496079744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761879.947523:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496079488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761879.947562:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496079616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761879.947579:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496079744:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761880.523342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761881.547313:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761882.571349:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761883.019544:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789761883.019606:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761883.019623:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761883.019667:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761883.019677:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761883.019686:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789761883.019728:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761879/real 1789761883] req@ffff8dd20504e680 x1876699496079488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.019755:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496079488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.019767:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.019774:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761883.019778:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496079488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761883.019786:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761879/real 1789761883] req@ffff8dd20504c700 x1876699496079616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.019803:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496079616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.019812:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.019815:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761883.019818:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496079616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761883.019824:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761879/real 1789761883] req@ffff8dd20504e300 x1876699496079744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.019836:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496079744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.019845:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.019848:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761883.019851:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496079744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761883.019858:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761879/real 1789761883] req@ffff8dd20504d180 x1876699496079360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761883.019869:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496079360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761883.019877:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761883.019880:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761883.019883:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496079360:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789761883.595328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761884.619302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761885.067252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761885.067268:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761885.067273:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761885.067287:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1864 00000100:00080000:1.0:1789761885.067302:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761885.067311:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761885.067349:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496079872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761885.067387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761885.067393:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761885.067395:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761885.067400:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1864 00000100:00080000:1.0:1789761885.067407:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761885.067413:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761885.067424:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761885.067441:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496080000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761885.067449:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496079872:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761885.067457:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761885.067464:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761885.067466:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761885.067470:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1864 00000100:00080000:1.0:1789761885.067473:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761885.067477:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761885.067482:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761885.067493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496080128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761885.067506:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761885.067512:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761885.067515:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761885.067520:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1864 00000100:00080000:1.0:1789761885.067536:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761885.067546:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761885.067547:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761885.067565:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761885.067582:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496080256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761885.067620:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761885/real 1789761885] req@ffff8dd20504d500 x1876699496079872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761885.067645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496079872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761885.067659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761885.067664:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761885.067669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496079872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761885.067702:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496080000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761885.067729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761885.067743:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761885/real 1789761885] req@ffff8dd20504df80 x1876699496080000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761885.067757:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496080000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761885.067766:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761885.067770:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761885.067774:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496080000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761885.067782:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496080128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761885.067805:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761885.067817:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761885/real 1789761885] req@ffff8dd20504fb80 x1876699496080128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761885.067839:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496080128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761885.067847:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761885.067850:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761885.067852:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496080128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761885.067859:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496080256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761885.067883:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761885.067896:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761885/real 1789761885] req@ffff8dd20504f800 x1876699496080256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761885.067909:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496080256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761885.067918:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761885.067922:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761885.067925:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496080256:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761885.643350:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761886.667338:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761887.691287:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761888.715338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761889.739303:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761890.187231:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761890.187240:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761890.187243:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761890.187252:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1869 00000100:00080000:1.0:1789761890.187261:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761890.187267:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761890.187297:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496080384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761890.187319:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761890.187321:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761890.187323:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761890.187326:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1869 00000100:00080000:1.0:1789761890.187328:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761890.187330:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761890.187338:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761890.187353:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496080512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761890.187364:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496080384:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761890.187367:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761890.187373:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761890.187375:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761890.187380:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1869 00000100:00080000:1.0:1789761890.187385:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761890.187389:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761890.187396:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761890.187407:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496080640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761890.187418:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761890.187422:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761890.187424:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761890.187428:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1869 00000100:00080000:1.0:1789761890.187434:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761890.187439:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761890.187444:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761890.187451:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761890.187464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496080768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761890.187488:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761890/real 1789761890] req@ffff8dd20504ce00 x1876699496080384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761890.187507:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496080384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761890.187515:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761890.187519:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761890.187521:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496080384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761890.187541:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496080512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761890.187559:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761890.187568:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761890/real 1789761890] req@ffff8dd20504f100 x1876699496080512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761890.187576:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496080512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761890.187582:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761890.187584:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761890.187586:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496080512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761890.187590:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496080640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761890.187601:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761890.187608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761890/real 1789761890] req@ffff8dd20504ed80 x1876699496080640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761890.187615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496080640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761890.187625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761890.187627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761890.187629:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496080640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761890.187632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496080768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761890.187646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761890.187656:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761890/real 1789761890] req@ffff8dd20504d880 x1876699496080768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761890.187667:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496080768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761890.187675:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761890.187678:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761890.187682:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496080768:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761890.763367:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761891.787324:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761892.811382:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761893.835322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761894.859380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761895.307317:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761895.307329:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761895.307334:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761895.307348:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1874 00000100:00080000:2.0:1789761895.307364:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761895.307373:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761895.307426:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496080896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761895.307461:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761895.307466:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761895.307468:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761895.307474:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1874 00000100:00080000:2.0:1789761895.307479:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761895.307484:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761895.307496:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761895.307509:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496080896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761895.307517:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496081024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761895.307534:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761895.307539:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761895.307540:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761895.307545:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1874 00000100:00080000:2.0:1789761895.307552:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761895.307557:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761895.307564:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761895.307580:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496081152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761895.307592:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761895.307600:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761895.307606:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761895.307608:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761895.307616:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1874 00000100:00080000:2.0:1789761895.307622:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761895.307628:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761895.307634:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761895.307639:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761895/real 1789761895] req@ffff8dd205264700 x1876699496080896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761895.307654:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496081280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761895.307657:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496080896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761895.307733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761895.307740:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761895.307746:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496080896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761895.307774:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496081024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761895.307795:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761895.307804:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761895/real 1789761895] req@ffff8dd205266a00 x1876699496081024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761895.307813:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496081024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761895.307819:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761895.307822:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761895.307824:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496081024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761895.307828:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496081152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761895.307840:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761895.307847:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761895/real 1789761895] req@ffff8dd205265880 x1876699496081152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761895.307854:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496081152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761895.307860:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761895.307862:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761895.307864:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496081152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761895.307869:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496081280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761895.307880:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761895.307886:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761895/real 1789761895] req@ffff8dd205264000 x1876699496081280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761895.307893:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496081280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761895.307898:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761895.307900:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761895.307902:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496081280:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761895.883365:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761896.907335:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761897.931362:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761898.955393:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761899.979350:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761900.427325:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761900.427335:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761900.427338:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761900.427347:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1880 00000100:00080000:2.0:1789761900.427358:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761900.427366:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761900.427407:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496081408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761900.427439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761900.427443:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761900.427445:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761900.427449:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1880 00000100:00080000:2.0:1789761900.427454:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761900.427458:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761900.427468:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761900.427483:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496081536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761900.427496:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761900.427498:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496081408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761900.427504:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761900.427509:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761900.427514:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1880 00000100:00080000:2.0:1789761900.427518:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761900.427523:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761900.427529:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761900.427543:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496081664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761900.427554:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761900.427558:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761900.427561:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761900.427564:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1880 00000100:00080000:2.0:1789761900.427568:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761900.427574:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761900.427578:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761900.427586:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761900.427601:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496081792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761900.427627:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761900/real 1789761900] req@ffff8dd205267b80 x1876699496081408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761900.427649:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496081408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761900.427660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761900.427665:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761900.427669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496081408:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761900.427696:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496081536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761900.427722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761900.427737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761900/real 1789761900] req@ffff8dd205267800 x1876699496081536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761900.427751:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496081536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761900.427761:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761900.427764:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761900.427768:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496081536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761900.427775:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496081664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761900.427797:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761900.427811:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761900/real 1789761900] req@ffff8dd205267100 x1876699496081664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761900.427824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496081664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761900.427833:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761900.427837:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761900.427840:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496081664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761900.427857:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496081792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761900.427883:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761900.427896:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761900/real 1789761900] req@ffff8dd205264e00 x1876699496081792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761900.427910:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496081792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761900.427919:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761900.427922:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761900.427926:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496081792:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761901.003331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761902.027334:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761903.051360:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761904.075343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761905.099317:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761905.547333:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761905.547343:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761905.547346:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761905.547354:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1885 00000100:00080000:3.0:1789761905.547363:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761905.547370:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761905.547403:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496081920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761905.547425:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761905.547428:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761905.547430:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761905.547435:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1885 00000100:00080000:3.0:1789761905.547439:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761905.547444:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761905.547454:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761905.547458:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496081920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761905.547472:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496082048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761905.547486:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761905.547490:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761905.547493:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761905.547497:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1885 00000100:00080000:3.0:1789761905.547510:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761905.547516:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761905.547522:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761905.547529:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761905.547539:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496082176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761905.547552:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761905.547555:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761905.547557:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761905.547562:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1885 00000100:00080000:3.0:1789761905.547567:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761905.547570:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761905.547575:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761905/real 1789761905] req@ffff8dd203145c00 x1876699496081920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789761905.547575:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761905.547590:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496082304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761905.547616:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496081920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761905.547626:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761905.547629:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761905.547632:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496081920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761905.547651:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496082048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761905.547670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761905.547679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761905/real 1789761905] req@ffff8dd203145180 x1876699496082048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761905.547688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496082048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761905.547694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761905.547696:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761905.547699:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496082048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761905.547703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496082176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761905.547715:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761905.547721:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761905/real 1789761905] req@ffff8dd203147800 x1876699496082176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761905.547729:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496082176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761905.547734:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761905.547736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761905.547738:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496082176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761905.547742:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496082304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761905.547751:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761905.547757:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761905/real 1789761905] req@ffff8dd203146300 x1876699496082304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761905.547764:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496082304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761905.547774:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761905.547775:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761905.547777:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496082304:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761906.123341:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761907.147346:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761908.171352:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761909.195319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761910.219334:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761910.667281:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761910.667291:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761910.667295:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761910.667304:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1890 00000100:00080000:3.0:1789761910.667314:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761910.667319:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761910.667352:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496082432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761910.667375:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761910.667378:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761910.667379:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761910.667382:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1890 00000100:00080000:3.0:1789761910.667385:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761910.667387:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761910.667394:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761910.667403:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496082560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761910.667415:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761910.667417:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761910.667419:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761910.667424:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1890 00000100:00100000:0.0:1789761910.667425:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496082432:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761910.667427:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761910.667433:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761910.667439:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761910.667445:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496082688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761910.667452:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761910.667454:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761910.667456:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761910.667458:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1890 00000100:00080000:3.0:1789761910.667460:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761910.667462:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761910.667466:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761910.667473:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496082816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761910.667498:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761910.667536:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761910/real 1789761910] req@ffff8dd203144380 x1876699496082432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761910.667548:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496082432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761910.667555:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761910.667558:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761910.667561:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496082432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761910.667579:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496082560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761910.667594:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761910.667602:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761910/real 1789761910] req@ffff8dd203144a80 x1876699496082560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761910.667610:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496082560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761910.667629:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761910.667631:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761910.667633:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496082560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761910.667638:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496082688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761910.667650:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761910.667656:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761910/real 1789761910] req@ffff8dd203144700 x1876699496082688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761910.667664:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496082688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761910.667669:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761910.667670:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761910.667672:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496082688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761910.667676:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496082816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761910.667685:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761910.667696:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761910/real 1789761910] req@ffff8dd203145f80 x1876699496082816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761910.667704:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496082816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761910.667710:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761910.667711:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761910.667714:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496082816:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761911.243347:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761912.267371:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761913.291341:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761914.315337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761915.339299:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761915.787297:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761915.787310:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761915.787315:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761915.787328:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1895 00000100:00080000:2.0:1789761915.787340:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761915.787348:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761915.787389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496082944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761915.787409:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761915.787413:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761915.787415:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761915.787420:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1895 00000100:00080000:2.0:1789761915.787430:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761915.787436:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761915.787440:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496082944:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789761915.787449:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761915.787466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496083072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761915.787475:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761915.787493:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761915.787495:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761915.787498:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1895 00000100:00080000:2.0:1789761915.787501:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761915.787507:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761915.787510:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789761915.787518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761915.787528:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496083200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761915.787536:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761915.787539:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761915.787540:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761915.787543:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1895 00000100:00080000:2.0:1789761915.787546:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761915.787548:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761915.787552:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761915.787556:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761915/real 1789761915] req@ffff8dd205264380 x1876699496082944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761915.787562:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496083328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761915.787580:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496082944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761915.787589:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761915.787593:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761915.787596:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496082944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761915.787616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496083072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761915.787634:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761915.787643:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761915/real 1789761915] req@ffff8dd205264a80 x1876699496083072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761915.787651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496083072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761915.787656:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761915.787659:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761915.787661:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496083072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761915.787667:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496083200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761915.787678:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761915.787684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761915/real 1789761915] req@ffff8dd205267480 x1876699496083200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761915.787692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496083200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761915.787697:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761915.787699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761915.787701:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496083200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761915.787704:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496083328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761915.787714:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761915.787720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761915/real 1789761915] req@ffff8dd205266680 x1876699496083328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761915.787727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496083328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761915.787732:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761915.787733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761915.787735:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496083328:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761916.363368:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761917.387358:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761918.411334:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761919.435350:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761920.459323:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761920.907307:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761920.907317:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761920.907321:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761920.907345:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1900 00000100:00080000:0.0:1789761920.907356:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761920.907362:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761920.907400:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496083456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761920.907445:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761920.907448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761920.907449:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761920.907452:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1900 00000100:00080000:0.0:1789761920.907455:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761920.907458:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761920.907466:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761920.907476:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496083584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761920.907484:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761920.907487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761920.907488:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761920.907491:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1900 00000100:00080000:0.0:1789761920.907494:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761920.907499:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496083456:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789761920.907500:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761920.907508:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761920.907519:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496083712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761920.907527:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761920.907529:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761920.907530:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761920.907532:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1900 00000100:00080000:0.0:1789761920.907534:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761920.907537:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789761920.907539:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761920.907544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496083840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761920.907633:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761920.907674:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761920/real 1789761920] req@ffff8dd211834700 x1876699496083456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761920.907692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496083456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761920.907701:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761920.907705:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761920.907708:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496083456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761920.907715:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496083584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761920.907734:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761920.907760:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761920/real 1789761920] req@ffff8dd211835c00 x1876699496083584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761920.907775:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496083584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761920.907781:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761920.907784:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761920.907802:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496083584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789761920.907859:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496083712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789761920.907908:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761920.907920:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761920/real 1789761920] req@ffff8dd211836a00 x1876699496083712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761920.907930:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496083712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761920.907936:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761920.907939:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761920.907941:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496083712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789761920.907946:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496083840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761920.907960:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761920.907967:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761920/real 1789761920] req@ffff8dd211834000 x1876699496083840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761920.907994:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496083840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761920.908009:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761920.908011:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761920.908014:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496083840:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761921.483374:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761922.507333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761923.531421:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761924.555303:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761925.579317:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761926.027344:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761926.027372:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761926.027376:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761926.027387:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1905 00000100:00080000:1.0:1789761926.027400:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761926.027407:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761926.027468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496083968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761926.027502:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761926.027523:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761926.027525:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761926.027529:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1905 00000100:00080000:1.0:1789761926.027532:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761926.027535:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761926.027546:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761926.027561:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496084096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761926.027590:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761926.027595:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761926.027597:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761926.027606:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496083968:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761926.027608:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1905 00000100:00080000:1.0:1789761926.027618:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761926.027624:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761926.027641:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761926.027663:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496084224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761926.027692:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761926.027695:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761926.027697:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761926.027701:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1905 00000100:00080000:1.0:1789761926.027709:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761926.027713:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761926.027718:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761926.027727:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789761926.027733:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496084352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761926.027814:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761926/real 1789761926] req@ffff8dd20504c380 x1876699496083968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761926.027856:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496083968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761926.027869:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761926.027873:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761926.027877:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496083968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761926.027904:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496084096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761926.027932:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761926.027946:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761926/real 1789761926] req@ffff8dd20504dc00 x1876699496084096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761926.027959:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496084096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761926.027968:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761926.027972:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761926.027976:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496084096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789761926.027983:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496084224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789761926.028001:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761926.028010:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761926/real 1789761926] req@ffff8dd20504c000 x1876699496084224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761926.028022:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496084224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761926.028031:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761926.028034:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761926.028038:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496084224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789761926.028043:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496084352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761926.028059:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761926.028069:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761926/real 1789761926] req@ffff8dd20504ea00 x1876699496084352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761926.028098:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496084352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761926.028106:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761926.028108:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761926.028111:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496084352:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761926.603356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761927.627359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761928.651307:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761929.675364:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761930.699351:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761931.147246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761931.147255:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761931.147258:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761931.147267:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1910 00000100:00080000:1.0:1789761931.147277:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761931.147282:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761931.147314:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496084480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761931.147338:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761931.147341:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761931.147342:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761931.147345:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1910 00000100:00080000:1.0:1789761931.147347:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761931.147350:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761931.147357:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761931.147367:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496084608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761931.147373:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761931.147375:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761931.147376:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761931.147378:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1910 00000100:00080000:1.0:1789761931.147380:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761931.147382:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761931.147385:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761931.147390:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496084736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761931.147397:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761931.147399:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761931.147401:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761931.147404:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1910 00000100:00080000:1.0:1789761931.147406:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761931.147408:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761931.147411:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761931.147416:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496084864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761931.147439:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496084480:192.168.203.109@tcp:250:kworker.0 00000800:00000100:2.0:1789761931.147506:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761931.147542:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761931/real 1789761931] req@ffff8dd20504ca80 x1876699496084480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761931.147555:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496084480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761931.147563:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761931.147566:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761931.147568:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496084480:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761931.147574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496084608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761931.147588:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761931.147596:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761931/real 1789761931] req@ffff8dd20504ea00 x1876699496084608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761931.147604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496084608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761931.147609:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761931.147611:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761931.147613:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496084608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789761931.147617:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496084736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789761931.147627:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761931.147633:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761931/real 1789761931] req@ffff8dd20504c000 x1876699496084736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761931.147645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496084736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761931.147650:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761931.147651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761931.147653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496084736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789761931.147657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496084864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761931.147666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761931.147672:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761931/real 1789761931] req@ffff8dd20504dc00 x1876699496084864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761931.147679:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496084864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761931.147684:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761931.147686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761931.147687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496084864:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761931.723383:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761932.749657:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761933.771389:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761934.795312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761935.819375:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761936.267270:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761936.267283:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761936.267288:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761936.267300:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1915 00000100:00080000:1.0:1789761936.267313:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761936.267321:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761936.267360:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496084992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761936.267397:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761936.267400:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761936.267402:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761936.267405:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1915 00000100:00080000:1.0:1789761936.267408:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761936.267411:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761936.267418:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761936.267428:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496085120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761936.267436:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761936.267440:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761936.267441:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761936.267444:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1915 00000100:00080000:1.0:1789761936.267446:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761936.267448:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761936.267452:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761936.267457:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496084992:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761936.267464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496085248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761936.267473:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761936.267475:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761936.267477:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761936.267479:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1915 00000100:00080000:1.0:1789761936.267481:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761936.267484:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761936.267488:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761936.267497:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496085376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761936.267532:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761936.267571:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761936/real 1789761936] req@ffff8dd20504c380 x1876699496084992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761936.267585:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496084992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761936.267594:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761936.267597:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761936.267600:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496084992:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761936.267605:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496085120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761936.267620:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761936.267628:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761936/real 1789761936] req@ffff8dd20504d880 x1876699496085120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761936.267636:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496085120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761936.267642:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761936.267644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761936.267646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496085120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789761936.267665:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496085248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789761936.267707:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761936.267719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761936/real 1789761936] req@ffff8dd20504ed80 x1876699496085248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761936.267732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496085248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761936.267739:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761936.267741:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761936.267743:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496085248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789761936.267748:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496085376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789761936.267763:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789761936.267770:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761936/real 1789761936] req@ffff8dd20504f100 x1876699496085376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761936.267778:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496085376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761936.267783:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761936.267785:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789761936.267787:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496085376:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761936.843351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761937.867348:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761938.891348:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761939.915480:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761940.939397:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761941.387322:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761941.387335:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761941.387339:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761941.387351:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1920 00000100:00080000:2.0:1789761941.387362:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761941.387369:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761941.387419:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496085504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761941.387460:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761941.387464:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761941.387467:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761941.387471:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1920 00000100:00080000:2.0:1789761941.387475:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761941.387481:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761941.387492:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761941.387502:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496085504:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789761941.387510:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496085632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761941.387520:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761941.387522:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761941.387524:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761941.387527:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1920 00000100:00080000:2.0:1789761941.387530:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761941.387533:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761941.387538:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761941.387546:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496085760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761941.387553:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761941.387558:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761941.387559:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761941.387562:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761941.387566:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1920 00000100:00080000:2.0:1789761941.387569:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761941.387571:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761941.387574:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761941.387581:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496085888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761941.387607:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761941/real 1789761941] req@ffff8dd205265500 x1876699496085504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761941.387624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496085504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761941.387643:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761941.387648:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761941.387653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496085504:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761941.387676:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496085632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761941.387701:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761941.387714:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761941/real 1789761941] req@ffff8dd205266680 x1876699496085632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761941.387726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496085632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761941.387734:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761941.387738:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761941.387741:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496085632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761941.387747:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496085760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761941.387765:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761941.387774:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761941/real 1789761941] req@ffff8dd205267480 x1876699496085760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761941.387792:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496085760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761941.387800:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761941.387803:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761941.387806:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496085760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761941.387811:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496085888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761941.387828:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761941.387838:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761941/real 1789761941] req@ffff8dd205264a80 x1876699496085888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761941.387848:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496085888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761941.387856:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761941.387859:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761941.387862:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496085888:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761941.963360:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761942.987305:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761944.011292:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761945.035328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761946.059326:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761946.507574:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761946.507587:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761946.507592:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761946.507613:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1926 00000100:00080000:2.0:1789761946.507622:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761946.507628:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761946.507661:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496086016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761946.507685:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761946.507688:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761946.507690:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761946.507692:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1926 00000100:00080000:2.0:1789761946.507695:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761946.507697:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761946.507705:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761946.507715:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496086144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761946.507726:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761946.507730:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761946.507732:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761946.507737:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1926 00000100:00080000:2.0:1789761946.507744:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761946.507746:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496086016:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761946.507750:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761946.507762:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761946.507776:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496086272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761946.507785:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761946.507787:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761946.507789:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761946.507791:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1926 00000100:00080000:2.0:1789761946.507794:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761946.507797:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761946.507801:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761946.507808:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496086400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761946.507901:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496086144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761946.507950:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496086272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761946.507985:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496086400:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761947.083354:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761948.107298:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761949.131339:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761949.579480:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789761949.579537:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761949.579559:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761949.579628:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761949.579641:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789761949.579651:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789761949.579710:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761946/real 1789761949] req@ffff8dd205264e00 x1876699496086144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761949.579743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496086144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761949.579757:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761949.579763:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761949.579768:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496086144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761949.579777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761946/real 1789761949] req@ffff8dd205267100 x1876699496086272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761949.579789:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496086272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761949.579798:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761949.579802:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761949.579806:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496086272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761949.579814:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761946/real 1789761949] req@ffff8dd205267800 x1876699496086400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761949.579828:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496086400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761949.579844:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761949.579849:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761949.579852:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496086400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761949.579861:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761946/real 1789761949] req@ffff8dd205264380 x1876699496086016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761949.579875:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496086016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761949.579884:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761949.579887:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761949.579891:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496086016:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789761950.155322:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761951.179340:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761951.627292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761951.627306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761951.627311:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761951.627325:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1931 00000100:00080000:2.0:1789761951.627339:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761951.627348:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789761951.627396:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496086528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761951.627443:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761951.627448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761951.627450:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761951.627455:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1931 00000100:00080000:2.0:1789761951.627460:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761951.627466:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761951.627481:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761951.627500:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496086656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761951.627512:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496086528:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789761951.627534:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761951.627541:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761951.627544:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761951.627553:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1931 00000100:00080000:2.0:1789761951.627573:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761951.627579:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761951.627588:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761951.627606:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496086784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761951.627620:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789761951.627626:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761951.627632:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761951.627635:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761951.627643:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1931 00000100:00080000:2.0:1789761951.627648:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761951.627655:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789761951.627665:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761951.627681:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496086912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761951.627694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761951/real 1789761951] req@ffff8dd205267b80 x1876699496086528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761951.627760:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496086528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761951.627774:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761951.627781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761951.627787:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496086528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761951.627818:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496086656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761951.627849:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761951.627863:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761951/real 1789761951] req@ffff8dd205264000 x1876699496086656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761951.627878:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496086656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761951.627887:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761951.627892:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761951.627895:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496086656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761951.627902:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496086784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761951.627930:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761951.627943:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761951/real 1789761951] req@ffff8dd205265880 x1876699496086784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761951.627956:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496086784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761951.627965:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761951.627969:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761951.627973:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496086784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761951.627979:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496086912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761951.628001:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761951.628013:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761951/real 1789761951] req@ffff8dd205266a00 x1876699496086912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761951.628026:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496086912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761951.628034:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761951.628037:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761951.628041:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496086912:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761952.203368:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761953.227343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761954.251339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761955.275371:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761956.299309:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761956.747258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761956.747269:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761956.747272:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761956.747288:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1936 00000100:00080000:3.0:1789761956.747298:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761956.747306:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761956.747344:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496087040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761956.747365:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761956.747368:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761956.747369:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761956.747373:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1936 00000100:00080000:3.0:1789761956.747377:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761956.747380:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761956.747389:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761956.747393:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496087040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761956.747403:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496087168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761956.747412:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761956.747415:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761956.747416:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761956.747420:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1936 00000100:00080000:3.0:1789761956.747422:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761956.747425:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761956.747430:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761956.747444:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496087296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761956.747450:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789761956.747460:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761956.747462:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761956.747464:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761956.747468:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1936 00000100:00080000:3.0:1789761956.747471:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761956.747473:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761956.747476:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761956.747486:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496087424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761956.747491:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761956/real 1789761956] req@ffff8dd203145500 x1876699496087040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761956.747515:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496087040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761956.747542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761956.747546:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761956.747549:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496087040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761956.747568:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496087168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761956.747588:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761956.747597:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761956/real 1789761956] req@ffff8dd203146a00 x1876699496087168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761956.747606:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496087168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761956.747612:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761956.747614:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761956.747616:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496087168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761956.747620:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496087296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761956.747631:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761956.747638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761956/real 1789761956] req@ffff8dd203144e00 x1876699496087296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761956.747645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496087296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761956.747650:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761956.747652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761956.747654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496087296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761956.747657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496087424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761956.747667:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761956.747673:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761956/real 1789761956] req@ffff8dd203146d80 x1876699496087424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761956.747681:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496087424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761956.747696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761956.747700:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761956.747704:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496087424:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761957.323345:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761958.347308:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761959.371321:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761960.395334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761961.419339:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761961.867260:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761961.867287:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761961.867290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761961.867300:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1941 00000100:00080000:3.0:1789761961.867309:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761961.867315:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761961.867368:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496087552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761961.867396:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761961.867399:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761961.867401:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761961.867405:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1941 00000100:00080000:3.0:1789761961.867408:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761961.867411:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761961.867419:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761961.867430:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496087680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761961.867439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761961.867442:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761961.867443:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761961.867446:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1941 00000100:00080000:3.0:1789761961.867449:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761961.867452:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761961.867456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761961.867458:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496087552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789761961.867467:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496087808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761961.867476:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761961.867494:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761961.867496:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761961.867501:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1941 00000100:00080000:3.0:1789761961.867504:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761961.867507:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761961.867512:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761961.867542:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496087936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761961.867566:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761961.867612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761961/real 1789761961] req@ffff8dd203147b80 x1876699496087552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761961.867629:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496087552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761961.867637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761961.867640:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761961.867642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496087552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761961.867649:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496087680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761961.867667:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761961.867677:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761961/real 1789761961] req@ffff8dd203146d80 x1876699496087680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761961.867685:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496087680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761961.867695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761961.867698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761961.867700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496087680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761961.867718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496087808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761961.867731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761961.867737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761961/real 1789761961] req@ffff8dd203144e00 x1876699496087808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761961.867744:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496087808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761961.867750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761961.867752:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761961.867753:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496087808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761961.867757:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496087936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761961.867767:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761961.867773:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761961/real 1789761961] req@ffff8dd203146a00 x1876699496087936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761961.867780:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496087936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761961.867785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761961.867787:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761961.867789:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496087936:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761962.443407:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761963.467297:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761964.491365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761965.515301:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761966.539315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761966.987277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761966.987286:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761966.987289:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761966.987298:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1946 00000100:00080000:1.0:1789761966.987307:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761966.987313:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761966.987357:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496088064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761966.987380:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761966.987383:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761966.987384:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761966.987387:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1946 00000100:00080000:1.0:1789761966.987389:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761966.987391:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761966.987399:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761966.987408:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496088192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761966.987415:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496088064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761966.987428:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761966.987431:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761966.987433:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761966.987438:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1946 00000100:00080000:1.0:1789761966.987453:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761966.987459:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761966.987465:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761966.987479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496088320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761966.987488:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761966.987496:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761966.987500:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761966.987502:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761966.987508:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1946 00000100:00080000:1.0:1789761966.987512:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761966.987516:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761966.987521:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761966.987536:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496088448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761966.987538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761966/real 1789761966] req@ffff8dd20504ce00 x1876699496088064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761966.987576:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496088064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761966.987586:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761966.987589:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761966.987592:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496088064:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761966.987610:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496088192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761966.987629:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761966.987638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761966/real 1789761966] req@ffff8dd20504f800 x1876699496088192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761966.987647:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496088192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761966.987652:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761966.987654:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761966.987657:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496088192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761966.987662:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496088320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761966.987675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761966.987685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761966/real 1789761966] req@ffff8dd20504fb80 x1876699496088320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761966.987698:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496088320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761966.987707:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761966.987710:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761966.987713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496088320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761966.987720:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496088448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761966.987748:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761966.987763:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761966/real 1789761966] req@ffff8dd20504df80 x1876699496088448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761966.987775:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496088448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761966.987784:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761966.987787:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761966.987791:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496088448:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761967.563306:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761968.587369:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761969.611365:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761970.635348:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761971.659332:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761972.107255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761972.107276:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761972.107280:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761972.107289:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1951 00000100:00080000:1.0:1789761972.107298:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761972.107303:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761972.107338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496088576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761972.107359:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761972.107362:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761972.107363:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761972.107366:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1951 00000100:00080000:1.0:1789761972.107369:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761972.107371:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761972.107379:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761972.107390:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496088704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761972.107398:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496088576:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789761972.107415:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761972.107421:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761972.107424:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761972.107428:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1951 00000100:00080000:1.0:1789761972.107433:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761972.107437:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761972.107444:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761972.107469:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496088832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761972.107479:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761972.107483:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761972.107485:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761972.107489:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1951 00000100:00080000:1.0:1789761972.107493:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761972.107494:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761972.107501:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761972.107508:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761972.107520:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496088960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761972.107536:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761972/real 1789761972] req@ffff8dd20504d500 x1876699496088576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761972.107552:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496088576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761972.107560:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761972.107562:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761972.107565:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496088576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761972.107585:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496088704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761972.107601:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761972.107610:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761972/real 1789761972] req@ffff8dd20504d180 x1876699496088704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761972.107618:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496088704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761972.107624:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761972.107626:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761972.107628:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496088704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761972.107632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496088832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761972.107644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761972.107650:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761972/real 1789761972] req@ffff8dd20504e300 x1876699496088832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761972.107662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496088832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761972.107667:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761972.107669:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761972.107671:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496088832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761972.107674:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496088960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761972.107684:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761972.107690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761972/real 1789761972] req@ffff8dd20504c700 x1876699496088960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761972.107697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496088960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761972.107702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761972.107704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761972.107706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496088960:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761972.683346:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761973.707299:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761974.731350:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761975.755383:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761976.779345:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761977.227288:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761977.227304:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761977.227310:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761977.227332:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1956 00000100:00080000:1.0:1789761977.227347:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761977.227356:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761977.227435:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496089088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761977.227470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761977.227475:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761977.227477:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761977.227482:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1956 00000100:00080000:1.0:1789761977.227487:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761977.227492:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761977.227504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761977.227507:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496089088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761977.227525:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496089216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761977.227539:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761977.227544:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761977.227546:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761977.227551:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1956 00000100:00080000:1.0:1789761977.227569:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761977.227575:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761977.227580:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789761977.227583:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761977.227601:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496089344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761977.227614:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761977.227619:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761977.227621:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761977.227625:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761977/real 1789761977] req@ffff8dd20504e680 x1876699496089088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761977.227630:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1956 00000100:00080000:1.0:1789761977.227639:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761977.227649:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761977.227657:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761977.227666:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496089088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761977.227673:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496089472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761977.227678:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761977.227687:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761977.227694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496089088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761977.227720:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496089216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761977.227739:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761977.227748:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761977/real 1789761977] req@ffff8dd20504c700 x1876699496089216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761977.227763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496089216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761977.227770:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761977.227772:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761977.227774:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496089216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761977.227778:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496089344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761977.227792:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761977.227799:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761977/real 1789761977] req@ffff8dd20504e300 x1876699496089344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761977.227807:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496089344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761977.227818:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761977.227820:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761977.227822:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496089344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761977.227825:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496089472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761977.227836:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761977.227841:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761977/real 1789761977] req@ffff8dd20504d180 x1876699496089472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761977.227848:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496089472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761977.227854:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761977.227856:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761977.227858:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496089472:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761977.803346:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761978.827311:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761979.851333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761980.875307:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761981.899312:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761982.347269:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761982.347281:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761982.347284:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761982.347292:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1961 00000100:00080000:1.0:1789761982.347301:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761982.347308:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761982.347355:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496089600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761982.347373:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761982.347376:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761982.347377:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761982.347380:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1961 00000100:00080000:1.0:1789761982.347383:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761982.347386:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761982.347393:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761982.347401:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496089600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761982.347405:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496089728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761982.347418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761982.347422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761982.347424:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761982.347429:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1961 00000100:00080000:1.0:1789761982.347433:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761982.347438:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761982.347444:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761982.347457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496089856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761982.347466:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789761982.347473:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761982.347476:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761982.347478:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761982.347482:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1961 00000100:00080000:1.0:1789761982.347485:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761982.347488:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761982.347492:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761982.347507:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761982/real 1789761982] req@ffff8dd20504d500 x1876699496089600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761982.347507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496089984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761982.347544:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496089600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761982.347553:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761982.347557:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761982.347560:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496089600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761982.347577:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496089728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761982.347592:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761982.347600:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761982/real 1789761982] req@ffff8dd20504df80 x1876699496089728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761982.347608:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496089728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761982.347613:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761982.347615:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761982.347618:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496089728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761982.347621:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496089856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761982.347632:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761982.347638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761982/real 1789761982] req@ffff8dd20504fb80 x1876699496089856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761982.347645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496089856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761982.347650:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761982.347652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761982.347653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496089856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761982.347657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496089984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761982.347666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761982.347672:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761982/real 1789761982] req@ffff8dd20504f800 x1876699496089984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761982.347679:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496089984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761982.347684:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761982.347686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761982.347688:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496089984:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761982.923342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761983.947326:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761984.971284:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761985.995334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761987.019319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761987.467285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761987.467308:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761987.467313:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761987.467323:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1967 00000100:00080000:1.0:1789761987.467333:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761987.467339:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789761987.467372:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496090112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761987.467394:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761987.467396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761987.467398:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761987.467401:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1967 00000100:00080000:1.0:1789761987.467403:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761987.467406:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761987.467419:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761987.467427:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496090112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789761987.467438:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496090240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761987.467451:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761987.467455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761987.467457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761987.467461:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1967 00000100:00080000:1.0:1789761987.467475:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761987.467479:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789761987.467483:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789761987.467486:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761987.467500:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496090368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761987.467508:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761987.467511:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761987.467512:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761987.467516:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1967 00000100:00100000:0.0:1789761987.467520:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761987/real 1789761987] req@ffff8dd20504ce00 x1876699496090112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761987.467521:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761987.467529:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789761987.467536:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761987.467544:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496090112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761987.467548:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496090496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761987.467568:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761987.467574:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761987.467577:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496090112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761987.467598:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496090240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761987.467613:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761987.467622:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761987/real 1789761987] req@ffff8dd20504f100 x1876699496090240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761987.467631:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496090240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761987.467637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761987.467639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761987.467641:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496090240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761987.467645:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496090368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761987.467655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761987.467661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761987/real 1789761987] req@ffff8dd20504ed80 x1876699496090368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761987.467672:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496090368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761987.467680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761987.467683:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761987.467687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496090368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761987.467693:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496090496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761987.467713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761987.467724:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761987/real 1789761987] req@ffff8dd20504d880 x1876699496090496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761987.467740:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496090496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761987.467750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761987.467752:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761987.467754:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496090496:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761988.043326:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761989.067338:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761990.091419:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761991.115388:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761992.139332:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761992.587317:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761992.587324:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761992.587328:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761992.587337:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1972 00000100:00080000:3.0:1789761992.587346:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761992.587352:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761992.587385:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496090624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761992.587410:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761992.587413:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761992.587414:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761992.587417:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1972 00000100:00080000:3.0:1789761992.587419:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761992.587421:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761992.587429:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761992.587438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496090752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761992.587446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761992.587449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761992.587450:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761992.587456:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496090624:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789761992.587457:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1972 00000100:00080000:3.0:1789761992.587463:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761992.587467:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761992.587470:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761992.587479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496090880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761992.587485:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761992.587487:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761992.587489:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761992.587491:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1972 00000100:00080000:3.0:1789761992.587494:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761992.587497:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761992.587500:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761992.587506:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496091008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761992.587515:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761992.587549:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761992/real 1789761992] req@ffff8dd203145500 x1876699496090624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761992.587563:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496090624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761992.587570:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761992.587573:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761992.587576:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496090624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761992.587594:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496090752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761992.587608:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761992.587616:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761992/real 1789761992] req@ffff8dd203145f80 x1876699496090752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761992.587640:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496090752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761992.587650:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761992.587652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761992.587654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496090752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761992.587658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496090880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761992.587671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761992.587677:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761992/real 1789761992] req@ffff8dd203144700 x1876699496090880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761992.587684:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496090880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761992.587689:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761992.587691:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761992.587693:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496090880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761992.587696:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496091008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761992.587706:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761992.587711:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761992/real 1789761992] req@ffff8dd203144a80 x1876699496091008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761992.587718:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496091008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761992.587723:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761992.587725:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761992.587727:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496091008:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789761993.163320:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761994.187336:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761995.211351:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761996.235418:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761997.259353:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761997.707286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761997.707296:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761997.707300:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761997.707311:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1977 00000100:00080000:3.0:1789761997.707323:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761997.707331:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789761997.707366:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496091136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761997.707394:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761997.707398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761997.707400:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761997.707406:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1977 00000100:00080000:3.0:1789761997.707410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761997.707414:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761997.707423:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496091136:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789761997.707424:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761997.707441:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496091264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761997.707453:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761997.707457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761997.707459:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761997.707464:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1977 00000100:00080000:3.0:1789761997.707468:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761997.707472:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789761997.707477:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761997.707487:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789761997.707491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496091392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761997.707502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761997.707506:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761997.707508:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761997.707512:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1977 00000100:00080000:3.0:1789761997.707516:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761997.707520:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789761997.707525:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761997/real 1789761997] req@ffff8dd203144380 x1876699496091136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789761997.707525:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761997.707540:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496091520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761997.707545:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496091136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761997.707568:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761997.707571:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761997.707574:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496091136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789761997.707593:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496091264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761997.707608:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761997.707616:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761997/real 1789761997] req@ffff8dd203146300 x1876699496091264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761997.707624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496091264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761997.707630:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761997.707631:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761997.707633:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496091264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789761997.707639:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496091392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789761997.707650:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761997.707656:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761997/real 1789761997] req@ffff8dd203147800 x1876699496091392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761997.707663:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496091392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761997.707669:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761997.707670:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761997.707672:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496091392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789761997.707675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496091520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789761997.707685:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789761997.707690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761997/real 1789761997] req@ffff8dd203145180 x1876699496091520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761997.707697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496091520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761997.707703:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761997.707704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789761997.707706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496091520:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789761998.283344:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761999.307366:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762000.331325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762001.355342:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762002.379310:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762002.827266:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762002.827275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762002.827278:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762002.827287:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1982 00000100:00080000:0.0:1789762002.827296:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762002.827302:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762002.827333:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496091648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762002.827355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762002.827358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762002.827360:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762002.827362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1982 00000100:00080000:0.0:1789762002.827365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762002.827367:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762002.827374:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762002.827383:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496091776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762002.827391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762002.827394:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762002.827397:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762002.827397:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496091648:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762002.827403:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1982 00000100:00080000:0.0:1789762002.827407:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762002.827410:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762002.827413:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762002.827420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496091904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762002.827427:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762002.827429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762002.827430:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762002.827432:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1982 00000100:00080000:0.0:1789762002.827434:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762002.827436:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762002.827439:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762002.827446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496092032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762002.827487:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762002.827521:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762002/real 1789762002] req@ffff8dd211835f80 x1876699496091648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762002.827534:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496091648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762002.827542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762002.827546:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762002.827549:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496091648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762002.827567:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496091776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762002.827582:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762002.827589:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762002/real 1789762002] req@ffff8dd211834380 x1876699496091776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762002.827597:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496091776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762002.827602:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762002.827604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762002.827606:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496091776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762002.827609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496091904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762002.827620:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762002.827626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762002/real 1789762002] req@ffff8dd211836680 x1876699496091904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762002.827632:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496091904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762002.827637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762002.827639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762002.827641:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496091904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762002.827645:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496092032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762002.827655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762002.827660:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762002/real 1789762002] req@ffff8dd211834a80 x1876699496092032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762002.827668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496092032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762002.827673:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762002.827674:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762002.827676:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496092032:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762003.403332:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762004.427333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762005.451329:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762006.475289:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762007.499294:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762007.947288:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762007.947299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762007.947304:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762007.947314:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1987 00000100:00080000:0.0:1789762007.947324:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762007.947331:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762007.947370:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496092160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762007.947392:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762007.947396:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762007.947399:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762007.947404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1987 00000100:00080000:0.0:1789762007.947409:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762007.947413:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762007.947420:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496092160:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789762007.947422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762007.947441:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496092288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762007.947453:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762007.947457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762007.947459:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762007.947464:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1987 00000100:00080000:0.0:1789762007.947468:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762007.947472:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762007.947478:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789762007.947485:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762007.947492:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496092416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762007.947504:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762007.947508:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762007.947510:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762007.947514:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1987 00000100:00080000:0.0:1789762007.947518:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762007.947522:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762007.947528:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762007.947528:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762007/real 1789762007] req@ffff8dd211835180 x1876699496092160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762007.947544:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496092544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762007.947554:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496092160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762007.947581:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762007.947586:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762007.947592:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496092160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762007.947615:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496092288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762007.947637:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762007.947650:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762007/real 1789762007] req@ffff8dd211837480 x1876699496092288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762007.947663:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496092288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762007.947672:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762007.947675:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762007.947679:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496092288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762007.947684:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496092416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762007.947713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762007.947724:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762007/real 1789762007] req@ffff8dd211834e00 x1876699496092416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762007.947737:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496092416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762007.947745:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762007.947748:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762007.947751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496092416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762007.947758:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496092544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762007.947773:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762007.947782:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762007/real 1789762007] req@ffff8dd211836d80 x1876699496092544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762007.947794:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496092544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762007.947803:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762007.947806:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762007.947809:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496092544:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762008.523304:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762009.547372:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762010.571333:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762011.595310:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762012.619342:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762013.067286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762013.067294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762013.067298:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762013.067307:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1992 00000100:00080000:1.0:1789762013.067316:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762013.067322:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762013.067354:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496092672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762013.067376:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762013.067379:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762013.067381:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762013.067383:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1992 00000100:00080000:1.0:1789762013.067390:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762013.067393:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762013.067401:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762013.067410:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496092800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762013.067418:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762013.067422:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762013.067424:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762013.067428:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1992 00000100:00080000:1.0:1789762013.067432:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762013.067437:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762013.067443:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496092672:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789762013.067445:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762013.067462:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496092928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762013.067476:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762013.067478:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762013.067480:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762013.067483:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1992 00000100:00080000:1.0:1789762013.067486:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762013.067489:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762013.067493:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762013.067503:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496093056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762013.067549:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496092800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762013.067609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496092928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762013.067640:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496093056:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762013.643320:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762014.667314:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762015.691314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789762016.139430:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789762016.139473:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789762016.139489:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762016.139528:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762016.139537:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762016.139546:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1789762016.139579:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762013/real 1789762016] req@ffff8dd20504c000 x1876699496092928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762016.139605:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496092928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762016.139616:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762016.139621:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762016.139625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496092928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762016.139633:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762013/real 1789762016] req@ffff8dd20504ea00 x1876699496093056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762016.139645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496093056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762016.139654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762016.139658:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762016.139661:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496093056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762016.139668:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762013/real 1789762016] req@ffff8dd20504c380 x1876699496092672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762016.139697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496092672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762016.139705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762016.139717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762016.139721:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496092672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762016.139728:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762013/real 1789762016] req@ffff8dd20504dc00 x1876699496092800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762016.139741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496092800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762016.139750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762016.139754:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762016.139757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496092800:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762016.715338:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762017.739324:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762018.187242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762018.187254:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762018.187259:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762018.187271:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 1997 00000100:00080000:1.0:1789762018.187281:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762018.187290:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762018.187324:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496093184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762018.187352:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762018.187356:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762018.187358:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762018.187363:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1997 00000100:00080000:1.0:1789762018.187367:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762018.187371:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762018.187381:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762018.187397:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496093312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762018.187412:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496093184:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762018.187414:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762018.187425:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762018.187431:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762018.187436:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 1997 00000100:00080000:1.0:1789762018.187439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762018.187445:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762018.187451:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762018.187464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496093440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762018.187487:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762018.187491:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762018.187493:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762018.187497:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 1997 00000100:00080000:1.0:1789762018.187502:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762018.187506:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762018.187511:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762018.187523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496093568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762018.187537:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762018.187599:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762018/real 1789762018] req@ffff8dd20504ca80 x1876699496093184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762018.187624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496093184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762018.187638:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762018.187644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762018.187648:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496093184:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762018.187695:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496093312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762018.187725:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762018.187740:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762018/real 1789762018] req@ffff8dd20504dc00 x1876699496093312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762018.187755:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496093312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762018.187764:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762018.187768:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762018.187771:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496093312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762018.187778:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496093440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762018.187804:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762018.187818:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762018/real 1789762018] req@ffff8dd20504c380 x1876699496093440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762018.187838:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496093440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762018.187854:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762018.187858:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762018.187862:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496093440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762018.187869:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496093568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762018.187893:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762018.187905:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762018/real 1789762018] req@ffff8dd20504ea00 x1876699496093568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762018.187917:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496093568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762018.187926:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762018.187929:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762018.187933:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496093568:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762018.763365:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762019.787296:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762020.811296:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762021.835309:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762022.859325:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762023.307262:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762023.307272:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762023.307277:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762023.307287:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2002 00000100:00080000:2.0:1789762023.307297:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762023.307304:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762023.307341:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496093696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762023.307376:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762023.307380:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762023.307382:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762023.307387:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2002 00000100:00080000:2.0:1789762023.307391:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762023.307396:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762023.307405:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762023.307411:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496093696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762023.307422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496093824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762023.307435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762023.307439:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762023.307442:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762023.307446:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2002 00000100:00080000:2.0:1789762023.307450:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762023.307454:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762023.307460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762023.307466:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762023.307475:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496093952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762023.307487:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762023.307491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762023.307493:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762023.307500:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762023/real 1789762023] req@ffff8dd205264700 x1876699496093696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762023.307501:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2002 00000100:00080000:2.0:1789762023.307513:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762023.307521:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762023.307528:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762023.307542:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496093696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762023.307544:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496094080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762023.307574:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762023.307579:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762023.307583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496093696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762023.307605:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496093824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762023.307625:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762023.307634:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762023/real 1789762023] req@ffff8dd205266300 x1876699496093824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762023.307643:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496093824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762023.307648:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762023.307651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762023.307655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496093824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762023.307661:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496093952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762023.307683:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762023.307695:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762023/real 1789762023] req@ffff8dd205265180 x1876699496093952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762023.307708:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496093952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762023.307715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762023.307718:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762023.307720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496093952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762023.307724:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496094080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762023.307739:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762023.307747:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762023/real 1789762023] req@ffff8dd205266d80 x1876699496094080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762023.307755:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496094080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762023.307760:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762023.307762:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762023.307764:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496094080:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762023.883289:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762024.907377:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762025.931355:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762026.955298:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762027.979302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762028.427263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762028.427271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762028.427274:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762028.427282:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2008 00000100:00080000:3.0:1789762028.427292:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762028.427298:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762028.427330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496094208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762028.427351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762028.427354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762028.427355:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762028.427358:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2008 00000100:00080000:3.0:1789762028.427360:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762028.427362:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762028.427369:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762028.427378:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496094336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762028.427386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762028.427389:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762028.427390:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762028.427393:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2008 00000100:00080000:3.0:1789762028.427395:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762028.427399:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762028.427404:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762028.427408:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496094208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762028.427413:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496094464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762028.427423:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762028.427432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762028.427434:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762028.427436:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2008 00000100:00080000:3.0:1789762028.427438:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762028.427441:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762028.427445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762028.427453:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496094592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762028.427477:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762028.427511:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762028/real 1789762028] req@ffff8dd203145c00 x1876699496094208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762028.427525:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496094208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762028.427533:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762028.427536:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762028.427540:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496094208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762028.427545:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496094336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762028.427561:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762028.427568:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762028/real 1789762028] req@ffff8dd203147480 x1876699496094336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762028.427581:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496094336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762028.427586:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762028.427588:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762028.427590:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496094336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762028.427607:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496094464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762028.427618:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762028.427624:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762028/real 1789762028] req@ffff8dd203144000 x1876699496094464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762028.427632:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496094464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762028.427637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762028.427639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762028.427640:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496094464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762028.427644:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496094592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762028.427653:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762028.427658:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762028/real 1789762028] req@ffff8dd203147100 x1876699496094592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762028.427665:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496094592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762028.427675:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762028.427677:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762028.427679:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496094592:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762029.003322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762030.027288:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762031.051299:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762032.075340:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762033.099459:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762033.547357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762033.547378:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762033.547382:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762033.547392:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2013 00000100:00080000:2.0:1789762033.547404:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762033.547411:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762033.547448:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496094720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762033.547477:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762033.547480:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762033.547482:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762033.547486:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2013 00000100:00080000:2.0:1789762033.547499:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762033.547503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762033.547513:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762033.547528:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496094848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762033.547550:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762033.547553:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762033.547555:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762033.547559:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496094720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762033.547560:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2013 00000100:00080000:2.0:1789762033.547568:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762033.547576:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762033.547582:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762033.547594:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496094976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762033.547604:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762033.547608:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762033.547610:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762033.547613:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2013 00000100:00080000:2.0:1789762033.547616:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762033.547620:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762033.547625:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762033.547634:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496095104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762033.547654:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762033.547707:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762033/real 1789762033] req@ffff8dd205265f80 x1876699496094720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762033.547732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496094720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762033.547760:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762033.547766:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762033.547772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496094720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762033.547802:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496094848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762033.547836:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762033.547866:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762033/real 1789762033] req@ffff8dd205266d80 x1876699496094848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762033.547885:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496094848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762033.547907:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762033.547913:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762033.547918:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496094848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762033.547927:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496094976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762033.547979:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762033.548002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762033/real 1789762033] req@ffff8dd205265180 x1876699496094976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762033.548019:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496094976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762033.548028:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762033.548031:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762033.548034:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496094976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762033.548040:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496095104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762033.548062:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762033.548072:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762033/real 1789762033] req@ffff8dd205266300 x1876699496095104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762033.548083:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496095104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762033.548091:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762033.548111:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762033.548115:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496095104:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762034.123379:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762035.147346:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762036.171330:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762037.195306:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762038.219445:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762038.667299:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762038.667313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762038.667319:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762038.667333:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2018 00000100:00080000:2.0:1789762038.667348:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762038.667357:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762038.667402:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496095232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762038.667445:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762038.667451:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762038.667454:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762038.667460:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2018 00000100:00080000:2.0:1789762038.667465:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762038.667470:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762038.667498:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762038.667515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496095360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762038.667531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762038.667536:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496095232:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762038.667538:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762038.667544:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762038.667552:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2018 00000100:00080000:2.0:1789762038.667557:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762038.667563:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762038.667571:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762038.667599:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496095488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762038.667613:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762038.667614:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762038.667622:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762038.667626:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762038.667634:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2018 00000100:00080000:2.0:1789762038.667640:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762038.667645:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762038.667652:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762038.667660:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762038/real 1789762038] req@ffff8dd205264700 x1876699496095232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762038.667672:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496095616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762038.667682:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496095232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762038.667702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762038.667707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762038.667710:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496095232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762038.667731:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496095360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762038.667748:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762038.667757:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762038/real 1789762038] req@ffff8dd205266a00 x1876699496095360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762038.667767:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496095360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762038.667774:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762038.667777:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762038.667780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496095360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762038.667785:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496095488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762038.667798:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762038.667805:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762038/real 1789762038] req@ffff8dd205265880 x1876699496095488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762038.667814:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496095488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762038.667820:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762038.667822:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762038.667825:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496095488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762038.667829:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496095616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762038.667843:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762038.667850:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762038/real 1789762038] req@ffff8dd205264000 x1876699496095616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762038.667859:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496095616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762038.667865:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762038.667867:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762038.667869:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496095616:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762039.243326:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762040.267394:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762041.291273:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762042.315345:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762043.339297:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762043.787224:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762043.787232:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762043.787236:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762043.787245:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2023 00000100:00080000:2.0:1789762043.787255:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762043.787263:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762043.787295:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496095744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762043.787318:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762043.787321:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762043.787322:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762043.787325:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2023 00000100:00080000:2.0:1789762043.787327:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762043.787330:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762043.787340:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762043.787352:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496095872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762043.787364:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762043.787369:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762043.787372:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762043.787377:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496095744:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762043.787382:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2023 00000100:00080000:2.0:1789762043.787395:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762043.787403:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762043.787409:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762043.787421:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496096000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762043.787429:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762043.787432:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762043.787433:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762043.787436:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2023 00000100:00080000:2.0:1789762043.787439:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762043.787442:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762043.787447:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762043.787454:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496096128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762043.787474:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762043.787521:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762043/real 1789762043] req@ffff8dd205267b80 x1876699496095744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762043.787537:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496095744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762043.787547:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762043.787551:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762043.787554:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496095744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762043.787580:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496095872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762043.787596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762043.787611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762043/real 1789762043] req@ffff8dd205264380 x1876699496095872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762043.787620:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496095872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762043.787626:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762043.787629:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762043.787631:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496095872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762043.787635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496096000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762043.787648:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762043.787654:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762043/real 1789762043] req@ffff8dd205267800 x1876699496096000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762043.787662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496096000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762043.787672:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762043.787674:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762043.787676:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496096000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762043.787679:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496096128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762043.787690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762043.787696:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762043/real 1789762043] req@ffff8dd205267100 x1876699496096128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762043.787703:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496096128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762043.787708:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762043.787709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762043.787711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496096128:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762044.363372:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762045.387367:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762046.411363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762047.435290:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762048.459315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762048.907237:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762048.907245:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762048.907249:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762048.907258:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2028 00000100:00080000:2.0:1789762048.907267:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762048.907273:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762048.907304:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496096256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762048.907326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762048.907329:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762048.907330:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762048.907333:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2028 00000100:00080000:2.0:1789762048.907335:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762048.907337:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762048.907351:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762048.907361:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496096384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762048.907369:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762048.907372:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762048.907375:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762048.907377:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496096256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762048.907380:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2028 00000100:00080000:2.0:1789762048.907387:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762048.907390:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762048.907393:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762048.907406:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496096512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762048.907414:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762048.907417:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762048.907418:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762048.907420:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2028 00000100:00080000:2.0:1789762048.907423:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762048.907425:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762048.907428:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762048.907436:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762048.907438:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496096640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762048.907472:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762048/real 1789762048] req@ffff8dd205264e00 x1876699496096256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762048.907487:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496096256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762048.907495:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762048.907498:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762048.907501:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496096256:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762048.907526:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496096384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762048.907553:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762048.907568:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762048/real 1789762048] req@ffff8dd205264a80 x1876699496096384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762048.907577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496096384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762048.907583:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762048.907586:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762048.907588:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496096384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762048.907592:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496096512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762048.907607:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762048.907614:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762048/real 1789762048] req@ffff8dd205267480 x1876699496096512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762048.907622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496096512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762048.907637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762048.907639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762048.907641:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496096512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762048.907645:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496096640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762048.907657:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762048.907664:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762048/real 1789762048] req@ffff8dd205266680 x1876699496096640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762048.907672:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496096640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762048.907677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762048.907678:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762048.907680:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496096640:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762049.483305:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762050.507336:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762051.531341:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762052.555295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762053.579415:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762054.027273:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762054.027283:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762054.027288:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762054.027297:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2033 00000100:00080000:2.0:1789762054.027307:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762054.027314:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762054.027350:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496096768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762054.027382:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762054.027386:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762054.027388:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762054.027393:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2033 00000100:00080000:2.0:1789762054.027397:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762054.027402:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762054.027412:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762054.027427:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496096768:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762054.027432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496096896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762054.027454:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762054.027458:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762054.027460:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762054.027464:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2033 00000800:00000100:1.0:1789762054.027480:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762054.027505:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762054.027511:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762054.027518:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762054/real 1789762054] req@ffff8dd205265500 x1876699496096768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762054.027518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762054.027537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496097024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762054.027538:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496096768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762054.027569:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762054.027572:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762054.027573:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762054.027581:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496096768:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762054.027582:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762054.027587:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762054.027594:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2033 00000100:00080000:2.0:1789762054.027598:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762054.027603:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762054.027605:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496096896:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1789762054.027612:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762054.027628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762054.027629:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496097152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762054.027640:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762054/real 1789762054] req@ffff8dd205266680 x1876699496096896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762054.027652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496096896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762054.027663:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762054.027667:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762054.027670:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496096896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762054.027675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496097024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762054.027686:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762054.027692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762054/real 1789762054] req@ffff8dd205267480 x1876699496097024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762054.027700:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496097024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762054.027706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762054.027707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762054.027709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496097024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762054.027718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496097152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762054.027729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762054.027735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762054/real 1789762054] req@ffff8dd205264a80 x1876699496097152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762054.027743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496097152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762054.027748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762054.027750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762054.027752:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496097152:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762054.603313:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762055.627328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762056.651302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762057.675342:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762058.699335:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762059.147324:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762059.147335:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762059.147340:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762059.147351:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2038 00000100:00080000:1.0:1789762059.147363:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762059.147371:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762059.147416:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496097280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762059.147447:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762059.147450:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762059.147452:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762059.147455:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2038 00000100:00080000:1.0:1789762059.147459:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762059.147463:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762059.147473:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762059.147491:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496097280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762059.147493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496097408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762059.147514:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762059.147519:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762059.147522:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762059.147528:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2038 00000100:00080000:1.0:1789762059.147534:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762059.147540:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762059.147545:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762059.147560:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496097536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762059.147562:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762059.147578:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762059.147581:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762059.147583:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762059.147588:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2038 00000100:00080000:1.0:1789762059.147592:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762059.147595:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762059.147600:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762059.147610:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762059/real 1789762059] req@ffff8dd20504c000 x1876699496097280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762059.147610:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496097664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762059.147693:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496097280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762059.147709:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762059.147715:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762059.147720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496097280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762059.147746:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496097408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762059.147788:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762059.147802:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762059/real 1789762059] req@ffff8dd20504d880 x1876699496097408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762059.147816:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496097408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762059.147826:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762059.147830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762059.147833:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496097408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762059.147840:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496097536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762059.147857:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762059.147864:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762059/real 1789762059] req@ffff8dd20504ed80 x1876699496097536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762059.147872:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496097536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762059.147878:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762059.147880:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762059.147882:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496097536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762059.147886:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496097664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762059.147897:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762059.147903:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762059/real 1789762059] req@ffff8dd20504f100 x1876699496097664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762059.147910:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496097664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762059.147915:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762059.147917:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762059.147919:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496097664:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762059.723358:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762060.747359:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762061.771405:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762062.795297:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762063.819323:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762064.267284:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762064.267291:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762064.267295:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762064.267304:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2043 00000100:00080000:2.0:1789762064.267312:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762064.267318:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762064.267350:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496097792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762064.267371:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762064.267374:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762064.267375:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762064.267378:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2043 00000100:00080000:2.0:1789762064.267381:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762064.267383:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762064.267394:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762064.267403:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496097920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762064.267411:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762064.267413:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762064.267415:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762064.267417:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2043 00000100:00080000:2.0:1789762064.267419:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762064.267422:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762064.267426:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762064.267436:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496098048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762064.267438:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496097792:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762064.267447:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762064.267450:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762064.267451:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762064.267453:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2043 00000100:00080000:2.0:1789762064.267455:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762064.267457:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762064.267460:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762064.267468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496098176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762064.267511:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762064.267551:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762064/real 1789762064] req@ffff8dd205264e00 x1876699496097792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762064.267569:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496097792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762064.267580:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762064.267584:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762064.267588:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496097792:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762064.267596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496097920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762064.267617:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762064.267629:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762064/real 1789762064] req@ffff8dd205267100 x1876699496097920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762064.267642:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496097920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762064.267651:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762064.267655:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762064.267658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496097920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762064.267681:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496098048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762064.267703:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762064.267729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762064/real 1789762064] req@ffff8dd205267800 x1876699496098048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762064.267743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496098048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762064.267751:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762064.267754:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762064.267757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496098048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762064.267763:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496098176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762064.267780:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762064.267790:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762064/real 1789762064] req@ffff8dd205264380 x1876699496098176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762064.267801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496098176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762064.267815:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762064.267818:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762064.267821:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496098176:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762064.843306:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762065.867363:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762066.891338:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762067.915319:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762068.939298:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762069.387221:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762069.387230:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762069.387233:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762069.387242:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2048 00000100:00080000:2.0:1789762069.387251:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762069.387257:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762069.387288:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496098304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762069.387308:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762069.387311:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762069.387312:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762069.387315:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2048 00000100:00080000:2.0:1789762069.387318:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762069.387321:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762069.387329:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762069.387338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496098432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762069.387346:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496098304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762069.387348:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762069.387355:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762069.387358:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762069.387361:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2048 00000100:00080000:2.0:1789762069.387363:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762069.387365:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762069.387369:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762069.387378:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496098560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762069.387385:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762069.387387:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762069.387388:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762069.387391:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2048 00000100:00080000:2.0:1789762069.387394:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762069.387397:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762069.387400:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762069.387406:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496098688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762069.387416:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762069.387455:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762069/real 1789762069] req@ffff8dd205267b80 x1876699496098304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762069.387469:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496098304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762069.387477:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762069.387480:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762069.387483:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496098304:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762069.387501:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496098432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762069.387516:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762069.387524:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762069/real 1789762069] req@ffff8dd205264000 x1876699496098432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762069.387532:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496098432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762069.387538:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762069.387540:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762069.387542:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496098432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762069.387546:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496098560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762069.387558:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762069.387564:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762069/real 1789762069] req@ffff8dd205265880 x1876699496098560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762069.387571:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496098560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762069.387576:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762069.387578:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762069.387580:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496098560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762069.387583:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496098688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762069.387604:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762069.387611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762069/real 1789762069] req@ffff8dd205266a00 x1876699496098688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762069.387622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496098688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762069.387630:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762069.387633:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762069.387637:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496098688:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762069.963400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762070.987318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762072.011375:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762073.035325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762074.059310:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762074.507305:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762074.507313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762074.507317:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762074.507325:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2054 00000100:00080000:1.0:1789762074.507335:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762074.507341:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762074.507372:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496098816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762074.507398:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762074.507402:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762074.507404:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762074.507409:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2054 00000100:00080000:1.0:1789762074.507412:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762074.507416:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762074.507426:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762074.507441:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496098944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762074.507444:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496098816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762074.507461:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762074.507464:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762074.507466:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762074.507470:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2054 00000100:00080000:1.0:1789762074.507473:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762074.507476:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762074.507481:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762074.507490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496099072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762074.507497:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762074.507500:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762074.507501:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762074.507503:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2054 00000100:00080000:1.0:1789762074.507505:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762074.507510:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762074.507511:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762074.507516:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762074.507523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496099200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762074.507557:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762074/real 1789762074] req@ffff8dd20504ce00 x1876699496098816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762074.507578:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496098816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762074.507590:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762074.507594:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762074.507598:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496098816:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762074.507622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496098944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762074.507666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762074.507681:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762074/real 1789762074] req@ffff8dd20504f800 x1876699496098944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762074.507695:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496098944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762074.507705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762074.507708:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762074.507712:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496098944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762074.507719:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496099072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762074.507742:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762074.507755:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762074/real 1789762074] req@ffff8dd20504fb80 x1876699496099072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762074.507769:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496099072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762074.507777:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762074.507781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762074.507785:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496099072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762074.507791:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496099200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762074.507814:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762074.507827:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762074/real 1789762074] req@ffff8dd20504df80 x1876699496099200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762074.507840:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496099200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762074.507848:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762074.507851:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762074.507855:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496099200:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762075.083318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762076.107321:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762077.131334:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762078.155359:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762079.179380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762079.627307:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762079.627324:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762079.627330:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762079.627345:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2059 00000100:00080000:1.0:1789762079.627361:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762079.627371:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762079.627437:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496099328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762079.627465:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762079.627469:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762079.627470:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762079.627474:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2059 00000100:00080000:1.0:1789762079.627477:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762079.627481:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762079.627508:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762079.627525:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496099456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762079.627526:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496099328:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762079.627551:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762079.627572:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762079.627574:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762079.627580:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2059 00000100:00080000:1.0:1789762079.627584:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762079.627588:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762079.627595:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762079.627622:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496099584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762079.627632:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762079.627656:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762079.627658:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762079.627663:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2059 00000100:00080000:1.0:1789762079.627668:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762079.627671:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762079.627677:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762079.627686:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496099456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762079.627694:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496099712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762079.627721:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496099584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762079.627754:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496099712:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762080.203334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762081.227331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762082.251314:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762082.699425:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789762082.699462:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762082.699476:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762082.699504:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762082.699511:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762082.699516:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789762082.699653:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762079/real 1789762082] req@ffff8dd20504c700 x1876699496099712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762082.699721:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496099712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762082.699736:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762082.699742:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762082.699747:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496099712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762082.699757:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762079/real 1789762082] req@ffff8dd20504d180 x1876699496099456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762082.699771:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496099456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762082.699780:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762082.699785:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762082.699790:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496099456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762082.699897:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762079/real 1789762082] req@ffff8dd20504e300 x1876699496099584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762082.699912:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496099584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762082.699920:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762082.699925:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762082.699928:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496099584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762082.699935:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762079/real 1789762082] req@ffff8dd20504d500 x1876699496099328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762082.699947:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496099328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762082.699954:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762082.699958:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762082.699961:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496099328:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789762083.275358:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762084.299404:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762084.747299:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762084.747307:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762084.747311:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762084.747319:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2064 00000100:00080000:2.0:1789762084.747328:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762084.747334:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762084.747364:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496099840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762084.747385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762084.747388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762084.747389:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762084.747392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2064 00000100:00080000:2.0:1789762084.747394:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762084.747396:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762084.747403:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762084.747413:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496099968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762084.747424:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762084.747428:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762084.747429:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762084.747434:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2064 00000100:00080000:2.0:1789762084.747437:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762084.747444:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762084.747445:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496099840:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789762084.747453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762084.747468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496100096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762084.747479:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762084.747483:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762084.747486:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762084.747490:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2064 00000100:00080000:2.0:1789762084.747494:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762084.747498:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762084.747504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762084.747516:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762084.747518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496100224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762084.747580:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762084/real 1789762084] req@ffff8dd205264700 x1876699496099840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762084.747605:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496099840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762084.747627:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762084.747633:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762084.747637:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496099840:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762084.747646:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496099968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762084.747671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762084.747685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762084/real 1789762084] req@ffff8dd205266300 x1876699496099968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762084.747699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496099968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762084.747708:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762084.747711:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762084.747715:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496099968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762084.747740:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496100096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762084.747768:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762084.747783:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762084/real 1789762084] req@ffff8dd205265180 x1876699496100096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762084.747795:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496100096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762084.747808:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762084.747812:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762084.747816:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496100096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762084.747822:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496100224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762084.747842:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762084.747854:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762084/real 1789762084] req@ffff8dd205266d80 x1876699496100224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762084.747865:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496100224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762084.747873:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762084.747876:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762084.747878:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496100224:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762085.323380:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762086.347295:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762087.371315:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762088.395290:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762089.419366:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762089.867251:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762089.867258:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762089.867262:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762089.867272:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2069 00000100:00080000:1.0:1789762089.867281:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762089.867287:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762089.867317:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496100352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762089.867341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762089.867345:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762089.867348:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762089.867353:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2069 00000100:00080000:1.0:1789762089.867357:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762089.867362:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762089.867372:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762089.867383:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496100352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762089.867387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496100480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762089.867398:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762089.867400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762089.867402:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762089.867405:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2069 00000100:00080000:1.0:1789762089.867408:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762089.867412:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762089.867417:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762089.867430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496100608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762089.867442:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762089.867445:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762089.867447:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762089.867453:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762089.867457:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2069 00000100:00080000:1.0:1789762089.867463:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762089.867468:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762089.867474:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762089.867485:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496100736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762089.867501:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762089/real 1789762089] req@ffff8dd20504e680 x1876699496100352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762089.867536:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496100352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762089.867549:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762089.867554:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762089.867559:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496100352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762089.867582:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496100480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762089.867619:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762089.867635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762089/real 1789762089] req@ffff8dd20504d500 x1876699496100480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762089.867669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496100480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762089.867680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762089.867684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762089.867689:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496100480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762089.867697:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496100608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762089.867724:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762089.867739:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762089/real 1789762089] req@ffff8dd20504e300 x1876699496100608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762089.867754:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496100608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762089.867764:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762089.867769:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762089.867773:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496100608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762089.867781:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496100736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762089.867821:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762089.867835:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762089/real 1789762089] req@ffff8dd20504d180 x1876699496100736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762089.867849:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496100736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762089.867857:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762089.867861:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762089.867880:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496100736:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762090.443329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762091.467293:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762092.491324:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762093.515327:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762094.539364:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762094.987319:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762094.987331:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762094.987335:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762094.987347:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2074 00000100:00080000:2.0:1789762094.987359:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762094.987367:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762094.987409:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496100864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762094.987444:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762094.987448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762094.987450:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762094.987454:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2074 00000100:00080000:2.0:1789762094.987460:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762094.987464:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762094.987475:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762094.987511:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496100864:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762094.987514:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496100992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762094.987531:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762094.987535:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762094.987538:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762094.987543:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2074 00000100:00080000:2.0:1789762094.987548:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762094.987553:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762094.987560:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762094.987572:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762094.987577:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496101120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762094.987591:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762094.987595:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762094.987597:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762094.987602:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2074 00000100:00080000:2.0:1789762094.987607:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762094.987612:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762094.987618:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762094.987621:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762094/real 1789762094] req@ffff8dd205265f80 x1876699496100864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762094.987634:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496101248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762094.987641:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496100864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762094.987663:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762094.987671:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762094.987678:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496100864:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762094.987702:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496100992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762094.987720:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762094.987729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762094/real 1789762094] req@ffff8dd205266d80 x1876699496100992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762094.987738:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496100992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762094.987744:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762094.987747:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762094.987750:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496100992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762094.987753:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496101120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762094.987788:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762094.987800:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762094/real 1789762094] req@ffff8dd205265180 x1876699496101120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762094.987821:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496101120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762094.987838:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762094.987844:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762094.987850:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496101120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762094.987861:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496101248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762094.987898:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762094.987920:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762094/real 1789762094] req@ffff8dd205266300 x1876699496101248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762094.987942:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496101248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762094.987953:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762094.987957:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762094.987960:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496101248:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762095.563348:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762096.587387:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762097.611544:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762098.635303:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762099.659349:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762100.107299:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762100.107307:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762100.107311:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762100.107319:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2079 00000100:00080000:1.0:1789762100.107329:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762100.107334:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762100.107378:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496101376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762100.107400:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762100.107403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762100.107404:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762100.107407:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2079 00000100:00080000:1.0:1789762100.107417:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762100.107422:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762100.107430:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496101376:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789762100.107430:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762100.107457:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496101504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762100.107468:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762100.107471:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762100.107473:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762100.107476:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2079 00000100:00080000:1.0:1789762100.107491:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762100.107495:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789762100.107500:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789762100.107501:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762100.107514:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496101632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762100.107522:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762100.107525:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762100.107526:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762100.107528:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2079 00000100:00080000:1.0:1789762100.107531:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762100.107534:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762100.107538:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762100.107541:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762100/real 1789762100] req@ffff8dd20504c700 x1876699496101376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762100.107549:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496101760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762100.107561:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496101376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762100.107570:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762100.107573:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762100.107577:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496101376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762100.107597:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496101504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762100.107613:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762100.107621:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762100/real 1789762100] req@ffff8dd20504df80 x1876699496101504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762100.107630:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496101504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762100.107636:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762100.107639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762100.107641:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496101504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762100.107644:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496101632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762100.107655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762100.107661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762100/real 1789762100] req@ffff8dd20504fb80 x1876699496101632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762100.107669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496101632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762100.107674:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762100.107676:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762100.107678:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496101632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762100.107681:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496101760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762100.107691:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762100.107697:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762100/real 1789762100] req@ffff8dd20504f800 x1876699496101760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762100.107704:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496101760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762100.107714:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762100.107716:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762100.107718:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496101760:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762100.683375:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762101.707319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762102.731323:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762103.755279:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762104.779324:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762105.227286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762105.227298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762105.227303:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762105.227316:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2084 00000100:00080000:1.0:1789762105.227329:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762105.227337:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762105.227384:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496101888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762105.227422:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762105.227426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762105.227428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762105.227432:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2084 00000100:00080000:1.0:1789762105.227436:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762105.227440:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762105.227451:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762105.227464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496102016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762105.227474:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496101888:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762105.227477:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762105.227492:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762105.227495:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762105.227500:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2084 00000100:00080000:1.0:1789762105.227502:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762105.227505:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762105.227510:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762105.227522:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496102144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762105.227530:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762105.227533:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762105.227534:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762105.227537:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2084 00000100:00080000:1.0:1789762105.227547:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762105.227552:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762105.227556:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762105.227557:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762105.227577:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496102272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762105.227607:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762105/real 1789762105] req@ffff8dd20504ce00 x1876699496101888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762105.227622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496101888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762105.227631:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762105.227634:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762105.227637:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496101888:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762105.227656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496102016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762105.227672:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762105.227680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762105/real 1789762105] req@ffff8dd20504f100 x1876699496102016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762105.227688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496102016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762105.227694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762105.227697:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762105.227699:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496102016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762105.227703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496102144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762105.227714:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762105.227720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762105/real 1789762105] req@ffff8dd20504ed80 x1876699496102144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762105.227727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496102144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762105.227732:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762105.227734:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762105.227736:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496102144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762105.227740:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496102272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762105.227751:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762105.227756:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762105/real 1789762105] req@ffff8dd20504d880 x1876699496102272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762105.227763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496102272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762105.227768:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762105.227770:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762105.227773:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496102272:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762105.803416:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762106.827322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762107.851376:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762108.875421:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762109.899445:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762110.347357:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762110.347416:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762110.347422:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762110.347437:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2089 00000100:00080000:2.0:1789762110.347468:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762110.347481:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762110.347539:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496102400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762110.347582:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762110.347586:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762110.347588:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762110.347593:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2089 00000100:00080000:2.0:1789762110.347598:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762110.347620:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762110.347634:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762110.347655:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496102528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762110.347674:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496102400:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762110.347701:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762110.347708:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762110.347710:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762110.347721:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2089 00000100:00080000:2.0:1789762110.347729:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762110.347735:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762110.347749:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762110.347767:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496102656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762110.347783:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762110.347799:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762110.347806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762110.347811:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762110.347821:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2089 00000100:00080000:2.0:1789762110.347830:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762110.347872:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762110.347880:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762110.347891:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762110/real 1789762110] req@ffff8dd205264700 x1876699496102400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762110.347900:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496102784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762110.347965:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496102400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762110.348000:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762110.348006:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762110.348011:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496102400:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762110.348043:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496102528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762110.348092:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762110.348107:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762110/real 1789762110] req@ffff8dd205266a00 x1876699496102528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762110.348119:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496102528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762110.348138:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762110.348141:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762110.348145:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496102528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762110.348151:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496102656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762110.348168:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762110.348177:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762110/real 1789762110] req@ffff8dd205265880 x1876699496102656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762110.348187:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496102656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762110.348231:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762110.348234:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762110.348237:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496102656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762110.348243:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496102784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762110.348284:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762110.348298:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762110/real 1789762110] req@ffff8dd205264000 x1876699496102784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762110.348316:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496102784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762110.348331:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762110.348337:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762110.348342:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496102784:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762110.923313:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762111.947326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762112.971379:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762113.995407:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762115.019296:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762115.467317:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762115.467347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762115.467354:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762115.467394:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2095 00000100:00080000:1.0:1789762115.467405:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762115.467411:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762115.467443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496102912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762115.467465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762115.467467:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762115.467469:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762115.467472:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2095 00000100:00080000:1.0:1789762115.467474:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762115.467476:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762115.467484:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762115.467493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496103040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762115.467500:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762115.467503:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762115.467504:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762115.467506:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2095 00000100:00080000:1.0:1789762115.467508:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762115.467511:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762115.467515:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762115.467519:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496102912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762115.467525:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496103168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762115.467533:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762115.467535:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762115.467536:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762115.467539:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2095 00000100:00080000:1.0:1789762115.467541:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762115.467543:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762115.467546:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762115.467553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496103296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762115.467577:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762115.467610:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762115/real 1789762115] req@ffff8dd20504c000 x1876699496102912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762115.467624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496102912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762115.467631:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762115.467643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762115.467646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496102912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762115.467652:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496103040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762115.467676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762115.467685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762115/real 1789762115] req@ffff8dd20504ea00 x1876699496103040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762115.467699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496103040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762115.467705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762115.467707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762115.467709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496103040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762115.467728:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496103168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762115.467744:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762115.467751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762115/real 1789762115] req@ffff8dd20504c380 x1876699496103168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762115.467759:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496103168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762115.467770:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762115.467772:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762115.467774:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496103168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762115.467779:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496103296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762115.467790:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762115.467796:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762115/real 1789762115] req@ffff8dd20504dc00 x1876699496103296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762115.467804:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496103296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762115.467809:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762115.467810:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762115.467812:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496103296:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762116.043340:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762117.067403:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762118.091336:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762119.115345:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762120.139434:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762120.587265:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762120.587275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762120.587279:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762120.587288:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2100 00000100:00080000:1.0:1789762120.587311:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762120.587318:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762120.587352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496103424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762120.587376:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762120.587379:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762120.587381:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762120.587385:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2100 00000100:00080000:1.0:1789762120.587388:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762120.587392:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762120.587400:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762120.587411:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496103424:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762120.587418:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496103552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762120.587436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762120.587440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762120.587444:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762120.587449:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2100 00000100:00080000:1.0:1789762120.587466:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762120.587473:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789762120.587476:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789762120.587486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762120.587505:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496103680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762120.587515:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762120.587518:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762120.587520:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762120.587526:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762120/real 1789762120] req@ffff8dd20504ca80 x1876699496103424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762120.587527:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2100 00000100:00080000:1.0:1789762120.587533:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762120.587541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762120.587551:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762120.587563:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496103808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762120.587569:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496103424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762120.587586:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762120.587592:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762120.587598:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496103424:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762120.587621:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496103552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762120.587644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762120.587655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762120/real 1789762120] req@ffff8dd20504dc00 x1876699496103552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762120.587665:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496103552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762120.587672:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762120.587674:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762120.587677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496103552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762120.587681:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496103680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762120.587695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762120.587702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762120/real 1789762120] req@ffff8dd20504c380 x1876699496103680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762120.587730:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496103680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762120.587755:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762120.587757:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762120.587760:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496103680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762120.587765:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496103808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762120.587781:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762120.587790:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762120/real 1789762120] req@ffff8dd20504ea00 x1876699496103808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762120.587799:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496103808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762120.587806:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762120.587808:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762120.587810:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496103808:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762121.163359:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762122.187471:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762123.211433:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762124.235402:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762125.259322:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762125.707355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762125.707373:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762125.707380:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762125.707399:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2105 00000100:00080000:1.0:1789762125.707419:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762125.707432:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762125.707513:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496103936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762125.707562:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762125.707568:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762125.707571:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762125.707588:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2105 00000100:00080000:1.0:1789762125.707594:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762125.707600:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762125.707615:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762125.707638:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496104064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762125.707655:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496103936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762125.707660:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762125.707671:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762125.707675:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762125.707683:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2105 00000100:00080000:1.0:1789762125.707688:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762125.707695:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762125.707704:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762125.707736:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496104192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762125.707753:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762125.707760:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762125.707763:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762125.707769:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2105 00000100:00080000:1.0:1789762125.707776:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762125.707782:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762125.707790:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762125.707800:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762125.707819:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496104320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762125.707851:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762125/real 1789762125] req@ffff8dd20504c000 x1876699496103936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762125.707875:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496103936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762125.707888:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762125.707893:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762125.707897:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496103936:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762125.707929:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496104064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762125.707956:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762125.707969:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762125/real 1789762125] req@ffff8dd20504d880 x1876699496104064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762125.707982:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496104064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762125.707990:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762125.707994:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762125.707997:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496104064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762125.708002:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496104192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762125.708019:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762125.708028:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762125/real 1789762125] req@ffff8dd20504ed80 x1876699496104192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762125.708039:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496104192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762125.708065:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762125.708068:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762125.708072:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496104192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762125.708077:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496104320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762125.708091:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762125.708099:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762125/real 1789762125] req@ffff8dd20504f100 x1876699496104320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762125.708110:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496104320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762125.708118:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762125.708121:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762125.708132:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496104320:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762126.283376:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762127.307305:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762128.331325:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762129.355380:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762130.379346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762130.827288:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762130.827296:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762130.827300:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762130.827308:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2110 00000100:00080000:2.0:1789762130.827318:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762130.827323:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762130.827365:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496104448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762130.827386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762130.827389:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762130.827391:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762130.827394:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2110 00000100:00080000:2.0:1789762130.827396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762130.827399:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762130.827407:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762130.827415:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496104448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762130.827419:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496104576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762130.827428:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762130.827430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762130.827431:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762130.827434:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2110 00000100:00080000:2.0:1789762130.827436:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762130.827439:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762130.827442:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762130.827450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496104704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762130.827457:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762130.827459:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762130.827460:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762130.827463:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2110 00000100:00080000:2.0:1789762130.827465:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762130.827467:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762130.827471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762130.827473:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762130.827492:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496104832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762130.827529:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762130/real 1789762130] req@ffff8dd205267b80 x1876699496104448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762130.827546:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496104448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762130.827554:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762130.827558:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762130.827561:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496104448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762130.827578:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496104576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762130.827610:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762130.827620:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762130/real 1789762130] req@ffff8dd205264380 x1876699496104576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762130.827628:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496104576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762130.827634:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762130.827636:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762130.827639:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496104576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762130.827643:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496104704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762130.827655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762130.827684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762130/real 1789762130] req@ffff8dd205267800 x1876699496104704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762130.827717:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496104704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762130.827724:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762130.827726:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762130.827729:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496104704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762130.827751:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496104832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762130.827766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762130.827774:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762130/real 1789762130] req@ffff8dd205267100 x1876699496104832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762130.827782:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496104832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762130.827788:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762130.827790:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762130.827793:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496104832:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762131.403413:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762132.427354:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762133.451381:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762134.475342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762135.499388:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762135.947274:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762135.947281:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762135.947284:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762135.947294:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2115 00000100:00080000:2.0:1789762135.947303:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762135.947310:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762135.947345:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496104960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762135.947371:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762135.947375:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762135.947383:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762135.947387:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2115 00000100:00080000:2.0:1789762135.947391:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762135.947396:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762135.947404:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762135.947416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496105088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762135.947421:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496104960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762135.947437:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762135.947440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762135.947441:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762135.947445:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2115 00000100:00080000:2.0:1789762135.947448:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762135.947450:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762135.947454:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762135.947463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496105216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762135.947470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762135.947473:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762135.947475:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762135.947477:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2115 00000100:00080000:2.0:1789762135.947479:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762135.947484:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762135.947484:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762135.947489:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762135.947499:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496105344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762135.947528:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762135/real 1789762135] req@ffff8dd205264e00 x1876699496104960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762135.947548:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496104960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762135.947559:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762135.947562:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762135.947565:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496104960:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762135.947584:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496105088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762135.947603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762135.947612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762135/real 1789762135] req@ffff8dd205264a80 x1876699496105088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762135.947625:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496105088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762135.947631:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762135.947633:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762135.947635:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496105088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762135.947638:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496105216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762135.947649:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762135.947655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762135/real 1789762135] req@ffff8dd205267480 x1876699496105216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762135.947662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496105216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762135.947668:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762135.947669:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762135.947671:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496105216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762135.947675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496105344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762135.947686:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762135.947691:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762135/real 1789762135] req@ffff8dd205266680 x1876699496105344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762135.947698:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496105344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762135.947707:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762135.947709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762135.947712:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496105344:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762136.523306:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762137.547394:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762138.571307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762139.595356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762140.619326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762141.067274:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762141.067282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762141.067286:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762141.067294:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2120 00000100:00080000:3.0:1789762141.067303:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762141.067309:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762141.067341:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496105472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762141.067362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762141.067365:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762141.067366:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762141.067369:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2120 00000100:00080000:3.0:1789762141.067372:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762141.067374:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762141.067382:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762141.067395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496105600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762141.067397:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496105472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762141.067409:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762141.067412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762141.067413:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762141.067415:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2120 00000100:00080000:3.0:1789762141.067417:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762141.067420:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762141.067423:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762141.067432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496105728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762141.067438:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762141.067441:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762141.067442:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762141.067445:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2120 00000100:00080000:3.0:1789762141.067448:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762141.067450:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762141.067453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762141.067461:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496105856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762141.067468:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762141.067510:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762141/real 1789762141] req@ffff8dd203145880 x1876699496105472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.067523:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496105472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.067531:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.067534:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762141.067537:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496105472:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762141.067567:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496105600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762141.067587:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762141.067597:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762141/real 1789762141] req@ffff8dd203147100 x1876699496105600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.067605:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496105600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.067611:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.067613:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762141.067615:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496105600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762141.067619:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496105728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762141.067631:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762141.067640:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762141/real 1789762141] req@ffff8dd203144000 x1876699496105728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.067648:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496105728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.067654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.067656:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762141.067658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496105728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762141.067661:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496105856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762141.067670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762141.067676:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762141/real 1789762141] req@ffff8dd203147480 x1876699496105856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.067683:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496105856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.067688:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.067690:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762141.067692:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496105856:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762141.643325:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762142.667285:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762143.691362:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762144.715347:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762145.739379:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762146.187329:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762146.187339:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762146.187343:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762146.187355:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2125 00000100:00080000:0.0:1789762146.187367:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762146.187374:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762146.187421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496105984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762146.187454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762146.187458:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762146.187460:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762146.187464:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2125 00000100:00080000:0.0:1789762146.187468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762146.187472:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762146.187483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762146.187499:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496106112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762146.187512:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762146.187520:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762146.187520:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496105984:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762146.187528:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762146.187537:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2125 00000100:00080000:0.0:1789762146.187541:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762146.187547:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762146.187553:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762146.187568:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496106240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762146.187585:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762146.187589:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762146.187592:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762146.187597:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2125 00000100:00080000:0.0:1789762146.187601:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762146.187606:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762146.187612:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762146.187626:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496106368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762146.187656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496106112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762146.187686:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496106240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762146.187704:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496106368:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762146.763293:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762147.787352:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762148.811288:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762149.259387:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789762149.259425:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762149.259438:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762149.259467:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762149.259474:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762149.259479:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789762149.259543:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762146/real 1789762149] req@ffff8dd211834e00 x1876699496106112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762149.259564:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496106112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762149.259578:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762149.259582:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762149.259585:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496106112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762149.259593:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762146/real 1789762149] req@ffff8dd211837480 x1876699496106240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762149.259605:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496106240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762149.259614:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762149.259617:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762149.259620:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496106240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762149.259627:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762146/real 1789762149] req@ffff8dd211835180 x1876699496106368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762149.259637:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496106368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762149.259646:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762149.259649:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762149.259652:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496106368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762149.259658:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762146/real 1789762149] req@ffff8dd211836d80 x1876699496105984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762149.259671:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496105984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762149.259679:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762149.259682:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762149.259686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496105984:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789762149.835292:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762150.859286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762151.307211:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762151.307219:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762151.307223:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762151.307232:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2130 00000100:00080000:0.0:1789762151.307241:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762151.307247:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762151.307275:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496106496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762151.307295:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762151.307298:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762151.307299:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762151.307302:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2130 00000100:00080000:0.0:1789762151.307304:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762151.307307:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762151.307316:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762151.307324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496106624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762151.307332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762151.307335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762151.307337:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762151.307339:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496106496:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762151.307341:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2130 00000100:00080000:0.0:1789762151.307345:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762151.307350:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762151.307354:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762151.307375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496106752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762151.307386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762151.307389:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762151.307391:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762151.307395:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2130 00000100:00080000:0.0:1789762151.307401:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762151.307405:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762151.307411:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762151.307412:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762151.307428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496106880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762151.307458:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762151/real 1789762151] req@ffff8dd211834a80 x1876699496106496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762151.307478:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496106496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762151.307490:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762151.307495:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762151.307498:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496106496:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762151.307523:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496106624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762151.307548:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762151.307563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762151/real 1789762151] req@ffff8dd211836680 x1876699496106624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762151.307577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496106624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762151.307586:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762151.307590:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762151.307594:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496106624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762151.307601:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496106752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762151.307633:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762151.307645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762151/real 1789762151] req@ffff8dd211834380 x1876699496106752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762151.307657:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496106752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762151.307665:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762151.307669:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762151.307677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496106752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762151.307683:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496106880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762151.307703:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762151.307715:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762151/real 1789762151] req@ffff8dd211835f80 x1876699496106880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762151.307726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496106880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762151.307733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762151.307736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762151.307740:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496106880:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762151.883311:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762152.907374:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762153.931338:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762154.955355:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762155.979315:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762156.427310:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762156.427324:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762156.427330:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762156.427344:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2136 00000100:00080000:2.0:1789762156.427358:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762156.427368:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762156.427418:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496107008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762156.427450:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762156.427454:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762156.427457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762156.427462:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2136 00000100:00080000:2.0:1789762156.427465:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762156.427470:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762156.427482:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762156.427502:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496107136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762156.427510:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496107008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762156.427519:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762156.427524:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762156.427526:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762156.427532:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2136 00000100:00080000:2.0:1789762156.427537:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762156.427542:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762156.427548:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762156.427562:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496107264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762156.427576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762156.427580:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762156.427582:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762156.427587:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2136 00000100:00080000:2.0:1789762156.427591:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762156.427597:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762156.427603:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762156.427606:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762156.427616:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496107392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762156.427660:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762156/real 1789762156] req@ffff8dd205265500 x1876699496107008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762156.427683:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496107008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762156.427695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762156.427700:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762156.427704:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496107008:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762156.427733:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496107136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762156.427758:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762156.427773:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762156/real 1789762156] req@ffff8dd205266680 x1876699496107136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762156.427787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496107136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762156.427797:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762156.427801:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762156.427805:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496107136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762156.427811:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496107264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762156.427833:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762156.427845:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762156/real 1789762156] req@ffff8dd205267480 x1876699496107264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762156.427858:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496107264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762156.427875:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762156.427879:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762156.427882:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496107264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762156.427888:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496107392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762156.427908:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762156.427920:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762156/real 1789762156] req@ffff8dd205264a80 x1876699496107392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762156.427934:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496107392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762156.427942:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762156.427945:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762156.427949:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496107392:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762157.003359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762158.027315:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762159.051338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762160.075283:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762161.099307:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762161.547320:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762161.547330:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762161.547335:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762161.547345:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2141 00000100:00080000:2.0:1789762161.547362:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762161.547369:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762161.547405:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496107520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762161.547430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762161.547433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762161.547435:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762161.547439:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2141 00000100:00080000:2.0:1789762161.547442:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762161.547445:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762161.547455:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762161.547468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496107648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762161.547478:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496107520:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762161.547483:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762161.547487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762161.547489:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762161.547494:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2141 00000100:00080000:2.0:1789762161.547499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762161.547503:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762161.547509:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762161.547520:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496107776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762161.547530:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762161.547536:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789762161.547537:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762161.547541:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762161.547547:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2141 00000100:00080000:2.0:1789762161.547552:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762161.547555:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762161.547560:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762161.547572:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496107904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762161.547579:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762161/real 1789762161] req@ffff8dd205264e00 x1876699496107520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762161.547603:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496107520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762161.547614:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762161.547618:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762161.547623:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496107520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762161.547648:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496107648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762161.547669:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762161.547679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762161/real 1789762161] req@ffff8dd205267100 x1876699496107648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762161.547687:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496107648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762161.547693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762161.547695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762161.547697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496107648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762161.547701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496107776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762161.547712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762161.547718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762161/real 1789762161] req@ffff8dd205267800 x1876699496107776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762161.547725:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496107776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762161.547731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762161.547733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762161.547734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496107776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762161.547738:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496107904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762161.547747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762161.547753:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762161/real 1789762161] req@ffff8dd205264380 x1876699496107904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762161.547761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496107904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762161.547766:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762161.547767:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762161.547769:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496107904:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762162.123342:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762163.147309:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762164.171418:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762165.195365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762166.219355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762166.667348:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762166.667374:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762166.667379:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762166.667391:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2146 00000100:00080000:1.0:1789762166.667427:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762166.667436:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762166.667480:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496108032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762166.667514:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762166.667519:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762166.667521:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762166.667526:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2146 00000100:00080000:1.0:1789762166.667529:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762166.667533:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762166.667545:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762166.667578:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496108160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762166.667589:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496108032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762166.667636:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762166.667643:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762166.667646:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762166.667655:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2146 00000100:00080000:1.0:1789762166.667687:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762166.667695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762166.667701:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762166.667711:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762166.667741:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496108288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762166.667764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762166.667771:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762166/real 1789762166] req@ffff8dd20504ce00 x1876699496108032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762166.667774:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762166.667780:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762166.667791:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2146 00000100:00080000:1.0:1789762166.667801:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762166.667811:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762166.667818:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762166.667829:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496108032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762166.667834:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496108416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762166.667893:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762166.667904:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762166.667913:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496108032:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762166.667988:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496108160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762166.668022:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762166.668040:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762166/real 1789762166] req@ffff8dd20504f800 x1876699496108160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762166.668078:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496108160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762166.668093:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762166.668100:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762166.668106:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496108160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762166.668117:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496108288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762166.668156:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762166.668178:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762166/real 1789762166] req@ffff8dd20504fb80 x1876699496108288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762166.668261:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496108288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762166.668272:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762166.668276:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762166.668296:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496108288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762166.668301:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496108416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762166.668322:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762166.668332:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762166/real 1789762166] req@ffff8dd20504df80 x1876699496108416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762166.668342:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496108416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762166.668397:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762166.668401:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762166.668404:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496108416:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762167.243401:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762168.267400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762169.291359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762170.315313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762171.339287:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762171.787260:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762171.787268:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762171.787272:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762171.787282:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2151 00000100:00080000:1.0:1789762171.787295:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762171.787302:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762171.787338:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496108544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762171.787359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762171.787361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762171.787363:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762171.787366:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2151 00000100:00080000:1.0:1789762171.787369:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762171.787372:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762171.787380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762171.787393:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496108672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762171.787396:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496108544:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762171.787414:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762171.787431:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762171.787432:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762171.787435:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2151 00000100:00080000:1.0:1789762171.787438:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762171.787440:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762171.787445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762171.787453:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496108800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762171.787460:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762171.787463:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762171.787465:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762171.787467:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2151 00000100:00080000:1.0:1789762171.787470:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762171.787475:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762171.787475:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762171.787482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762171.787489:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496108928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762171.787524:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762171/real 1789762171] req@ffff8dd20504c700 x1876699496108544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762171.787545:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496108544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762171.787564:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762171.787569:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762171.787573:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496108544:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762171.787598:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496108672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762171.787623:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762171.787638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762171/real 1789762171] req@ffff8dd20504d180 x1876699496108672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762171.787652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496108672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762171.787661:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762171.787665:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762171.787668:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496108672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762171.787675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496108800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762171.787698:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762171.787720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762171/real 1789762171] req@ffff8dd20504e300 x1876699496108800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762171.787733:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496108800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762171.787742:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762171.787746:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762171.787749:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496108800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762171.787756:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496108928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762171.787776:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762171.787789:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762171/real 1789762171] req@ffff8dd20504d500 x1876699496108928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762171.787801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496108928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762171.787810:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762171.787813:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762171.787817:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496108928:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762172.363409:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762173.387312:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762174.411400:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762175.435330:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762176.459330:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762176.907281:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762176.907294:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762176.907299:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762176.907312:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2156 00000100:00080000:1.0:1789762176.907325:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762176.907334:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762176.907381:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496109056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762176.907418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762176.907423:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762176.907426:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762176.907431:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2156 00000100:00080000:1.0:1789762176.907435:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762176.907440:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762176.907452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762176.907468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496109184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762176.907497:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762176.907500:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762176.907501:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762176.907504:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2156 00000100:00080000:1.0:1789762176.907505:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762176.907508:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762176.907512:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762176.907518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496109312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762176.907526:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762176.907531:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789762176.907532:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496109056:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762176.907533:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762176.907541:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2156 00000100:00080000:1.0:1789762176.907544:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762176.907547:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762176.907550:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762176.907559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496109440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762176.907591:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762176.907629:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762176/real 1789762176] req@ffff8dd20504e680 x1876699496109056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762176.907647:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496109056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762176.907658:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762176.907662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762176.907666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496109056:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762176.907673:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496109184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762176.907696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762176.907710:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762176/real 1789762176] req@ffff8dd20504d500 x1876699496109184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762176.907724:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496109184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762176.907733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762176.907736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762176.907738:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496109184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762176.907757:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496109312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762176.907772:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762176.907779:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762176/real 1789762176] req@ffff8dd20504e300 x1876699496109312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762176.907787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496109312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762176.907793:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762176.907795:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762176.907797:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496109312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762176.907800:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496109440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762176.907809:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762176.907815:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762176/real 1789762176] req@ffff8dd20504d180 x1876699496109440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762176.907822:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496109440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762176.907831:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762176.907833:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762176.907835:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496109440:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762177.483319:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762178.507280:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762179.531291:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762180.555302:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762181.579294:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762182.027337:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762182.027352:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762182.027359:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762182.027374:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2161 00000100:00080000:1.0:1789762182.027393:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762182.027429:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762182.027508:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496109568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762182.027548:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762182.027553:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762182.027556:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762182.027561:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2161 00000100:00080000:1.0:1789762182.027566:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762182.027571:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762182.027597:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762182.027615:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496109696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762182.027633:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762182.027635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496109568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762182.027638:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762182.027645:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762182.027667:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2161 00000100:00080000:1.0:1789762182.027672:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762182.027676:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762182.027682:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762182.027699:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496109824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762182.027712:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762182.027718:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762182.027720:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762182.027725:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2161 00000100:00080000:1.0:1789762182.027760:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762182.027765:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762182.027771:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762182.027781:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762182.027789:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496109952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762182.027898:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762182/real 1789762182] req@ffff8dd20504c700 x1876699496109568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762182.027938:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496109568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762182.027960:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762182.027968:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762182.027977:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496109568:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762182.028022:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496109696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762182.028061:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762182.028080:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762182/real 1789762182] req@ffff8dd20504df80 x1876699496109696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762182.028113:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496109696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762182.028131:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762182.028136:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762182.028141:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496109696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762182.028150:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496109824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762182.028208:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762182.028225:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762182/real 1789762182] req@ffff8dd20504fb80 x1876699496109824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762182.028241:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496109824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762182.028252:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762182.028256:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762182.028260:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496109824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762182.028268:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496109952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762182.028290:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762182.028302:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762182/real 1789762182] req@ffff8dd20504f800 x1876699496109952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762182.028317:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496109952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762182.028327:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762182.028330:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762182.028334:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496109952:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762182.603477:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762183.627330:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762184.651309:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762185.675293:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762186.699307:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762187.147263:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762187.147271:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762187.147275:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762187.147283:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2166 00000100:00080000:2.0:1789762187.147293:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762187.147299:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762187.147329:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496110080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762187.147347:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762187.147351:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762187.147353:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762187.147357:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2166 00000100:00080000:2.0:1789762187.147361:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762187.147364:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762187.147390:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762187.147406:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496110080:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762187.147406:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496110208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762187.147419:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762187.147422:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762187.147424:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762187.147428:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2166 00000100:00080000:2.0:1789762187.147431:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762187.147436:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762187.147441:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762187.147452:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496110336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762187.147463:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762187.147466:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762187.147468:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762187.147472:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2166 00000100:00080000:2.0:1789762187.147475:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762187.147479:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762187.147484:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762187.147485:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762187.147494:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496110464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762187.147535:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762187/real 1789762187] req@ffff8dd205267b80 x1876699496110080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762187.147557:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496110080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762187.147566:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762187.147569:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762187.147572:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496110080:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762187.147592:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496110208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762187.147613:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762187.147625:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762187/real 1789762187] req@ffff8dd205264000 x1876699496110208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762187.147634:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496110208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762187.147641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762187.147643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762187.147646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496110208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762187.147650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496110336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762187.147663:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762187.147669:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762187/real 1789762187] req@ffff8dd205265880 x1876699496110336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762187.147678:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496110336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762187.147689:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762187.147691:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762187.147693:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496110336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762187.147697:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496110464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762187.147709:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762187.147716:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762187/real 1789762187] req@ffff8dd205266a00 x1876699496110464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762187.147724:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496110464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762187.147730:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762187.147732:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762187.147734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496110464:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762187.723282:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762188.747293:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762189.771305:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762190.795282:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762191.819295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762192.267263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762192.267271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762192.267275:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762192.267284:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2171 00000100:00080000:2.0:1789762192.267293:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762192.267299:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762192.267330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496110592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762192.267352:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762192.267354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762192.267356:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762192.267359:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2171 00000100:00080000:2.0:1789762192.267361:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762192.267363:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762192.267371:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762192.267379:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496110720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762192.267393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762192.267394:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496110592:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762192.267396:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762192.267400:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762192.267407:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2171 00000100:00080000:2.0:1789762192.267410:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762192.267414:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762192.267421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762192.267436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496110848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762192.267447:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762192.267450:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762192.267452:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762192.267456:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2171 00000800:00000100:0.0:1789762192.267461:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762192.267464:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762192.267469:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762192.267474:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762192.267484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496110976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762192.267505:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762192/real 1789762192] req@ffff8dd205264700 x1876699496110592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762192.267518:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496110592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762192.267525:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762192.267528:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762192.267531:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496110592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762192.267549:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496110720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762192.267566:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762192.267574:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762192/real 1789762192] req@ffff8dd205266300 x1876699496110720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762192.267583:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496110720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762192.267588:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762192.267590:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762192.267593:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496110720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762192.267596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496110848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762192.267607:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762192.267612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762192/real 1789762192] req@ffff8dd205265180 x1876699496110848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762192.267619:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496110848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762192.267625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762192.267626:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762192.267628:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496110848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762192.267632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496110976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762192.267642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762192.267648:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762192/real 1789762192] req@ffff8dd205266d80 x1876699496110976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762192.267655:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496110976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762192.267660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762192.267662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762192.267664:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496110976:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762192.843344:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762193.867423:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762194.891316:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762195.915324:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762196.939275:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762197.387264:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762197.387274:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762197.387277:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762197.387287:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2176 00000100:00080000:1.0:1789762197.387298:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762197.387306:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762197.387346:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496111104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762197.387386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762197.387390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762197.387392:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762197.387397:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2176 00000100:00080000:1.0:1789762197.387401:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762197.387405:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762197.387415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762197.387430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496111232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762197.387444:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762197.387448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762197.387451:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762197.387455:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2176 00000100:00080000:1.0:1789762197.387460:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762197.387467:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762197.387468:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496111104:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789762197.387475:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762197.387493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496111360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762197.387507:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762197.387512:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762197.387514:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762197.387518:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2176 00000100:00080000:1.0:1789762197.387524:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762197.387528:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789762197.387533:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789762197.387537:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762197.387564:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496111488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762197.387572:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762197/real 1789762197] req@ffff8dd20504ce00 x1876699496111104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762197.387593:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496111104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762197.387601:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762197.387604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762197.387608:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496111104:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762197.387614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496111232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762197.387628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762197.387636:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762197/real 1789762197] req@ffff8dd20504f100 x1876699496111232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762197.387644:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496111232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762197.387655:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762197.387657:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762197.387659:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496111232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762197.387677:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496111360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762197.387688:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762197.387694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762197/real 1789762197] req@ffff8dd20504ed80 x1876699496111360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762197.387702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496111360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762197.387707:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762197.387709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762197.387711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496111360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762197.387715:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496111488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762197.387746:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762197.387753:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762197/real 1789762197] req@ffff8dd20504d880 x1876699496111488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762197.387761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496111488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762197.387772:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762197.387775:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762197.387778:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496111488:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762197.963326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762198.987303:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762200.011343:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762201.035357:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762202.059326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762202.507269:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762202.507277:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762202.507280:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762202.507288:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2182 00000100:00080000:2.0:1789762202.507297:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762202.507304:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762202.507340:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496111616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762202.507358:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762202.507361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762202.507362:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762202.507365:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2182 00000100:00080000:2.0:1789762202.507367:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762202.507370:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762202.507378:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762202.507388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496111744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762202.507393:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496111616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762202.507402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762202.507405:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762202.507407:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762202.507410:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2182 00000100:00080000:2.0:1789762202.507413:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762202.507415:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762202.507419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762202.507428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496111872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762202.507434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762202.507437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762202.507439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762202.507442:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2182 00000100:00080000:2.0:1789762202.507444:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762202.507446:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762202.507450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762202.507451:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762202.507460:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496112000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762202.507498:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762202/real 1789762202] req@ffff8dd205265f80 x1876699496111616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762202.507512:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496111616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762202.507521:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762202.507524:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762202.507528:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496111616:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762202.507546:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496111744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762202.507563:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762202.507571:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762202/real 1789762202] req@ffff8dd205266d80 x1876699496111744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762202.507579:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496111744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762202.507585:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762202.507587:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762202.507589:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496111744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762202.507593:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496111872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762202.507604:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762202.507611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762202/real 1789762202] req@ffff8dd205265180 x1876699496111872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762202.507619:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496111872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762202.507624:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762202.507626:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762202.507628:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496111872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762202.507631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496112000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762202.507641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762202.507647:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762202/real 1789762202] req@ffff8dd205266300 x1876699496112000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762202.507655:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496112000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762202.507660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762202.507661:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762202.507663:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496112000:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762203.083329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762204.107307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762205.131289:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762206.155398:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762207.179340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762207.627290:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762207.627299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762207.627303:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762207.627313:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2187 00000100:00080000:3.0:1789762207.627323:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762207.627331:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762207.627369:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496112128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762207.627393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762207.627396:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762207.627397:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762207.627400:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2187 00000100:00080000:3.0:1789762207.627404:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762207.627407:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762207.627415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762207.627425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496112256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762207.627433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762207.627436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762207.627437:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762207.627442:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2187 00000100:00080000:3.0:1789762207.627445:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762207.627446:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496112128:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762207.627450:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762207.627458:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762207.627465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496112384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762207.627472:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762207.627475:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762207.627476:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762207.627479:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2187 00000100:00080000:3.0:1789762207.627481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762207.627483:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762207.627486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762207.627494:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496112512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762207.627515:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762207.627562:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762207/real 1789762207] req@ffff8dd203145c00 x1876699496112128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762207.627587:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496112128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762207.627596:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762207.627599:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762207.627604:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496112128:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762207.627622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496112256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762207.627658:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762207.627680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762207/real 1789762207] req@ffff8dd203145180 x1876699496112256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762207.627689:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496112256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762207.627695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762207.627698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762207.627700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496112256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762207.627704:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496112384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762207.627718:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762207.627724:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762207/real 1789762207] req@ffff8dd203147800 x1876699496112384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762207.627732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496112384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762207.627737:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762207.627739:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762207.627741:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496112384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762207.627745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496112512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762207.627755:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762207.627761:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762207/real 1789762207] req@ffff8dd203146300 x1876699496112512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762207.627768:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496112512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762207.627773:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762207.627775:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762207.627777:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496112512:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762208.203311:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762209.227300:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762210.251314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762211.275508:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762212.299310:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762212.747287:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762212.747298:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762212.747303:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762212.747316:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2192 00000100:00080000:3.0:1789762212.747338:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762212.747347:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762212.747395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496112640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762212.747423:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762212.747427:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762212.747429:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762212.747433:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2192 00000100:00080000:3.0:1789762212.747436:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762212.747439:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762212.747450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762212.747462:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496112768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762212.747472:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496112640:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762212.747475:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762212.747482:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762212.747487:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762212.747491:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2192 00000100:00080000:3.0:1789762212.747494:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762212.747496:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762212.747502:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762212.747513:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496112896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762212.747523:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762212.747526:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762212.747527:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762212.747530:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2192 00000100:00080000:3.0:1789762212.747533:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762212.747535:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762212.747539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762212.747550:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496113024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762212.747657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496112768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762212.747699:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496112896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762212.747719:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496113024:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762213.323317:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762214.347305:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762215.371397:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762215.819497:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789762215.819553:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762215.819575:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762215.819642:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762215.819654:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762215.819664:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789762215.819782:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762212/real 1789762215] req@ffff8dd203144a80 x1876699496112768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762215.819831:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496112768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762215.819848:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762215.819854:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762215.819860:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496112768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762215.819870:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762212/real 1789762215] req@ffff8dd203144700 x1876699496112896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762215.819885:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496112896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762215.819893:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762215.819897:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762215.819900:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496112896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762215.819908:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762212/real 1789762215] req@ffff8dd203145f80 x1876699496113024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762215.819921:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496113024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762215.819929:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762215.819933:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762215.819937:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496113024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762215.819945:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762212/real 1789762215] req@ffff8dd203144380 x1876699496112640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762215.819969:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496112640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762215.819979:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762215.819983:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762215.819987:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496112640:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789762216.395327:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762217.419345:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762217.867299:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762217.867310:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762217.867314:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762217.867324:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2197 00000100:00080000:3.0:1789762217.867336:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762217.867343:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762217.867378:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496113152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762217.867407:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762217.867412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762217.867413:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762217.867418:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2197 00000100:00080000:3.0:1789762217.867422:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762217.867426:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762217.867435:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762217.867451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496113280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762217.867457:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496113152:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762217.867470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762217.867474:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762217.867476:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762217.867482:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2197 00000100:00080000:3.0:1789762217.867487:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762217.867492:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762217.867499:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762217.867513:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496113408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762217.867527:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762217.867528:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762217.867532:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762217.867534:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762217.867539:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2197 00000100:00080000:3.0:1789762217.867545:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762217.867550:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762217.867556:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762217.867587:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762217/real 1789762217] req@ffff8dd203145500 x1876699496113152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762217.867589:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496113536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762217.867630:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496113152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762217.867644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762217.867649:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762217.867653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496113152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762217.867676:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496113280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762217.867702:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762217.867717:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762217/real 1789762217] req@ffff8dd203146a00 x1876699496113280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762217.867730:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496113280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762217.867739:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762217.867742:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762217.867746:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496113280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762217.867753:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496113408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762217.867775:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762217.867789:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762217/real 1789762217] req@ffff8dd203144e00 x1876699496113408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762217.867802:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496113408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762217.867811:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762217.867815:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762217.867818:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496113408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762217.867824:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496113536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762217.867846:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762217.867858:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762217/real 1789762217] req@ffff8dd203146d80 x1876699496113536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762217.867872:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496113536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762217.867881:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762217.867884:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762217.867887:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496113536:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762218.443345:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762219.467335:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762220.491315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762221.515346:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762222.539318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762222.987271:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762222.987280:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762222.987283:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762222.987292:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2202 00000100:00080000:2.0:1789762222.987301:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762222.987308:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762222.987347:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496113664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762222.987365:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762222.987368:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762222.987369:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762222.987372:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2202 00000100:00080000:2.0:1789762222.987374:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762222.987376:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762222.987384:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762222.987393:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496113792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762222.987400:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762222.987404:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762222.987407:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496113664:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762222.987407:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762222.987414:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2202 00000100:00080000:2.0:1789762222.987417:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762222.987420:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762222.987423:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762222.987432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496113920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762222.987439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762222.987442:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762222.987443:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762222.987445:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2202 00000100:00080000:2.0:1789762222.987448:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762222.987450:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762222.987453:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762222.987460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496114048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762222.987465:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762222.987506:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762222/real 1789762222] req@ffff8dd205264700 x1876699496113664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762222.987523:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496113664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762222.987544:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762222.987548:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762222.987551:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496113664:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762222.987568:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496113792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762222.987587:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762222.987596:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762222/real 1789762222] req@ffff8dd205266a00 x1876699496113792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762222.987604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496113792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762222.987610:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762222.987612:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762222.987614:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496113792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762222.987618:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496113920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762222.987629:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762222.987635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762222/real 1789762222] req@ffff8dd205265880 x1876699496113920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762222.987643:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496113920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762222.987648:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762222.987650:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762222.987652:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496113920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762222.987656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496114048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762222.987665:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762222.987671:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762222/real 1789762222] req@ffff8dd205264000 x1876699496114048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762222.987678:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496114048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762222.987684:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762222.987685:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762222.987687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496114048:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762223.563378:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762224.587292:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762225.611321:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762226.635283:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762227.659294:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762228.107301:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762228.107309:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762228.107313:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762228.107321:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2207 00000100:00080000:1.0:1789762228.107330:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762228.107336:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762228.107366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496114176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762228.107387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762228.107390:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762228.107392:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762228.107394:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2207 00000100:00080000:1.0:1789762228.107396:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762228.107398:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762228.107406:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762228.107414:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496114304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762228.107423:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762228.107427:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762228.107429:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762228.107432:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496114176:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762228.107437:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2207 00000100:00080000:1.0:1789762228.107443:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762228.107447:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762228.107453:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762228.107469:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496114432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762228.107481:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762228.107485:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762228.107503:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762228.107509:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762228.107511:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2207 00000100:00080000:1.0:1789762228.107517:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762228.107522:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762228.107528:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762228.107544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496114560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762228.107556:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762228/real 1789762228] req@ffff8dd20504c000 x1876699496114176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762228.107577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496114176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762228.107588:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762228.107592:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762228.107595:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496114176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762228.107614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496114304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762228.107634:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762228.107643:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762228/real 1789762228] req@ffff8dd20504ea00 x1876699496114304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762228.107652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496114304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762228.107658:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762228.107660:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762228.107662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496114304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762228.107666:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496114432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762228.107677:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762228.107683:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762228/real 1789762228] req@ffff8dd20504c380 x1876699496114432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762228.107691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496114432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762228.107696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762228.107697:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762228.107699:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496114432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762228.107703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496114560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762228.107713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762228.107719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762228/real 1789762228] req@ffff8dd20504dc00 x1876699496114560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762228.107726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496114560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762228.107736:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762228.107738:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762228.107740:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496114560:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762228.683292:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762229.707291:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762230.731278:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762231.755287:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762232.779366:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762233.227272:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762233.227282:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762233.227286:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762233.227297:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2212 00000100:00080000:1.0:1789762233.227308:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762233.227315:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762233.227351:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496114688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762233.227377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762233.227381:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762233.227383:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762233.227387:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2212 00000100:00080000:1.0:1789762233.227391:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762233.227395:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762233.227404:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762233.227412:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496114688:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762233.227420:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496114816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762233.227432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762233.227436:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762233.227438:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762233.227442:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2212 00000100:00080000:1.0:1789762233.227470:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762233.227473:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762233.227478:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762233.227479:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762233.227491:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496114944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762233.227500:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762233.227503:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762233.227504:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762233.227507:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2212 00000100:00080000:1.0:1789762233.227510:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762233.227512:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762233.227521:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762233/real 1789762233] req@ffff8dd20504ca80 x1876699496114688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762233.227521:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762233.227532:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496115072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762233.227541:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496114688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762233.227550:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762233.227553:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762233.227556:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496114688:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762233.227577:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496114816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762233.227593:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762233.227602:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762233/real 1789762233] req@ffff8dd20504dc00 x1876699496114816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762233.227610:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496114816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762233.227616:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762233.227618:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762233.227620:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496114816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762233.227625:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496114944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762233.227636:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762233.227642:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762233/real 1789762233] req@ffff8dd20504c380 x1876699496114944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762233.227650:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496114944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762233.227656:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762233.227657:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762233.227659:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496114944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762233.227663:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496115072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762233.227673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762233.227679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762233/real 1789762233] req@ffff8dd20504ea00 x1876699496115072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762233.227686:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496115072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762233.227692:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762233.227693:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762233.227695:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496115072:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762233.803290:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762234.827340:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762235.851273:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762236.875505:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762237.899484:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762238.347369:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762238.347384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762238.347390:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762238.347405:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2217 00000100:00080000:2.0:1789762238.347452:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762238.347463:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762238.347533:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496115200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762238.347575:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762238.347580:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762238.347582:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762238.347587:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2217 00000100:00080000:2.0:1789762238.347591:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762238.347596:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762238.347610:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762238.347629:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496115200:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762238.347631:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496115328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762238.347650:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762238.347654:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762238.347657:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762238.347662:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2217 00000100:00080000:2.0:1789762238.347666:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762238.347683:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762238.347691:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762238.347706:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496115456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762238.347723:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762238.347735:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762238.347740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762238.347743:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762238.347749:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2217 00000100:00080000:2.0:1789762238.347754:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762238.347759:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762238.347766:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762238.347781:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496115584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762238.347792:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762238/real 1789762238] req@ffff8dd205267b80 x1876699496115200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762238.347821:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496115200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762238.347862:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762238.347868:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762238.347873:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496115200:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762238.347903:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496115328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762238.347932:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762238.347948:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762238/real 1789762238] req@ffff8dd205264380 x1876699496115328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762238.347958:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496115328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762238.347965:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762238.347968:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762238.347970:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496115328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762238.347974:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496115456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762238.347991:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762238.347998:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762238/real 1789762238] req@ffff8dd205267800 x1876699496115456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762238.348006:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496115456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762238.348016:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762238.348030:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762238.348033:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496115456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762238.348036:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496115584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762238.348046:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762238.348052:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762238/real 1789762238] req@ffff8dd205267100 x1876699496115584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762238.348060:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496115584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762238.348066:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762238.348068:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762238.348069:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496115584:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762238.923327:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762239.947485:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762240.971347:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762241.995316:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762243.019281:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762243.467229:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762243.467237:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762243.467241:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762243.467250:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2223 00000100:00080000:2.0:1789762243.467259:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762243.467264:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762243.467295:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496115712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762243.467321:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762243.467324:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762243.467325:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762243.467328:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2223 00000100:00080000:2.0:1789762243.467331:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762243.467333:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762243.467341:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762243.467352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496115840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762243.467361:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762243.467363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762243.467365:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762243.467384:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2223 00000100:00100000:1.0:1789762243.467386:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496115712:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762243.467387:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762243.467394:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762243.467399:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762243.467427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496115968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762243.467436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762243.467439:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762243.467440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762243.467443:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2223 00000100:00080000:2.0:1789762243.467447:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762243.467449:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762243.467452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762243.467463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496116096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762243.467469:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762243.467508:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762243/real 1789762243] req@ffff8dd205264e00 x1876699496115712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762243.467527:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496115712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762243.467535:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762243.467538:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762243.467541:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496115712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762243.467547:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496115840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762243.467562:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762243.467571:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762243/real 1789762243] req@ffff8dd205264a80 x1876699496115840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762243.467579:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496115840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762243.467585:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762243.467587:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762243.467589:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496115840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762243.467608:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496115968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762243.467638:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762243.467645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762243/real 1789762243] req@ffff8dd205267480 x1876699496115968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762243.467652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496115968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762243.467662:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762243.467664:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762243.467666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496115968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762243.467670:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496116096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762243.467679:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762243.467700:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762243/real 1789762243] req@ffff8dd205266680 x1876699496116096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762243.467712:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496116096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762243.467720:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762243.467724:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762243.467727:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496116096:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762244.043358:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762245.067431:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762246.091267:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762247.115303:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762248.139310:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762248.587282:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762248.587291:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762248.587295:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762248.587306:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2228 00000100:00080000:0.0:1789762248.587316:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762248.587324:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762248.587372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496116224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762248.587404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762248.587408:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762248.587410:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762248.587414:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2228 00000100:00080000:0.0:1789762248.587418:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762248.587421:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762248.587430:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762248.587442:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496116352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762248.587455:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762248.587455:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496116224:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762248.587461:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762248.587463:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762248.587471:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2228 00000100:00080000:0.0:1789762248.587492:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762248.587495:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762248.587500:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762248.587510:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496116480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762248.587518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762248.587521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762248.587522:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762248.587524:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2228 00000100:00080000:0.0:1789762248.587529:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762248.587532:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762248.587536:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762248.587540:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762248.587549:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496116608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762248.587579:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762248/real 1789762248] req@ffff8dd211834000 x1876699496116224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762248.587593:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496116224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762248.587617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762248.587620:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762248.587623:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496116224:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762248.587645:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496116352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762248.587669:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762248.587682:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762248/real 1789762248] req@ffff8dd211836a00 x1876699496116352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762248.587696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496116352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762248.587705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762248.587709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762248.587712:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496116352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762248.587719:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496116480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762248.587759:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762248.587771:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762248/real 1789762248] req@ffff8dd211835c00 x1876699496116480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762248.587780:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496116480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762248.587786:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762248.587788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762248.587790:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496116480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762248.587794:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496116608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762248.587806:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762248.587813:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762248/real 1789762248] req@ffff8dd211834700 x1876699496116608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762248.587831:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496116608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762248.587837:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762248.587839:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762248.587841:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496116608:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762249.163297:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762250.187355:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762251.211345:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762252.235358:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762253.259334:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762253.707235:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762253.707246:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762253.707251:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762253.707260:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2233 00000100:00080000:0.0:1789762253.707269:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762253.707275:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762253.707319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496116736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762253.707340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762253.707342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762253.707344:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762253.707347:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2233 00000100:00080000:0.0:1789762253.707350:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762253.707352:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762253.707360:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762253.707371:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496116864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762253.707383:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762253.707387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762253.707388:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762253.707393:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496116736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762253.707394:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2233 00000100:00080000:0.0:1789762253.707398:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762253.707404:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762253.707408:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762253.707416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496116992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762253.707423:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762253.707426:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762253.707427:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762253.707429:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2233 00000100:00080000:0.0:1789762253.707431:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762253.707434:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762253.707437:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762253.707443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496117120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762253.707461:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762253.707504:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762253/real 1789762253] req@ffff8dd211837b80 x1876699496116736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762253.707525:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496116736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762253.707534:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762253.707537:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762253.707540:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496116736:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762253.707559:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496116864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762253.707578:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762253.707587:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762253/real 1789762253] req@ffff8dd211837100 x1876699496116864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762253.707595:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496116864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762253.707605:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762253.707607:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762253.707610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496116864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762253.707614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496116992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762253.707636:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762253.707642:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762253/real 1789762253] req@ffff8dd211836300 x1876699496116992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762253.707649:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496116992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762253.707655:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762253.707656:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762253.707658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496116992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762253.707662:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496117120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762253.707672:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762253.707678:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762253/real 1789762253] req@ffff8dd211837800 x1876699496117120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762253.707685:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496117120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762253.707690:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762253.707692:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762253.707694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496117120:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762254.283280:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762255.307383:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762256.331303:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762257.355290:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762258.379321:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762258.827367:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762258.827382:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762258.827389:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762258.827405:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2238 00000100:00080000:0.0:1789762258.827435:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762258.827443:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762258.827486:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496117248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762258.827517:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762258.827521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762258.827523:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762258.827527:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2238 00000100:00080000:0.0:1789762258.827530:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762258.827533:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762258.827543:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762258.827555:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496117376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762258.827566:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762258.827570:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762258.827572:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762258.827579:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2238 00000100:00080000:0.0:1789762258.827581:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762258.827585:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762258.827589:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496117248:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789762258.827595:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762258.827615:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496117504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762258.827632:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762258.827652:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762258.827654:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762258.827658:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2238 00000100:00080000:0.0:1789762258.827665:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762258.827669:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789762258.827675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789762258.827677:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762258.827689:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496117632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762258.827721:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762258/real 1789762258] req@ffff8dd211837800 x1876699496117248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762258.827735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496117248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762258.827743:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762258.827746:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762258.827749:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496117248:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762258.827768:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496117376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762258.827784:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762258.827819:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762258/real 1789762258] req@ffff8dd211836300 x1876699496117376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762258.827830:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496117376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762258.827838:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762258.827840:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762258.827843:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496117376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762258.827848:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496117504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762258.827865:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762258.827874:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762258/real 1789762258] req@ffff8dd211837100 x1876699496117504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762258.827884:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496117504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762258.827890:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762258.827892:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762258.827895:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496117504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762258.827899:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496117632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762258.827912:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762258.827919:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762258/real 1789762258] req@ffff8dd211837b80 x1876699496117632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762258.827928:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496117632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762258.827950:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762258.827952:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762258.827954:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496117632:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762259.403429:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762260.427355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762261.451339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762262.475417:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762263.499351:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762263.947257:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762263.947267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762263.947271:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762263.947282:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2243 00000100:00080000:0.0:1789762263.947292:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762263.947297:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762263.947330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496117760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762263.947352:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762263.947354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762263.947356:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762263.947359:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2243 00000100:00080000:0.0:1789762263.947361:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762263.947364:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762263.947371:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762263.947382:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496117888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762263.947391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762263.947394:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762263.947398:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762263.947399:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496117760:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762263.947403:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2243 00000100:00080000:0.0:1789762263.947408:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762263.947411:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762263.947414:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762263.947422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496118016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762263.947431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762263.947434:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762263.947435:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762263.947438:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2243 00000100:00080000:0.0:1789762263.947440:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762263.947442:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762263.947445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762263.947455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496118144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762263.947470:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762263.947506:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762263/real 1789762263] req@ffff8dd211834700 x1876699496117760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762263.947530:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496117760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762263.947542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762263.947546:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762263.947551:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496117760:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762263.947577:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496117888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762263.947603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762263.947619:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762263/real 1789762263] req@ffff8dd211835c00 x1876699496117888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762263.947634:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496117888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762263.947641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762263.947644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762263.947646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496117888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762263.947651:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496118016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762263.947668:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762263.947676:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762263/real 1789762263] req@ffff8dd211836a00 x1876699496118016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762263.947684:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496118016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762263.947694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762263.947696:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762263.947699:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496118016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762263.947702:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496118144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762263.947712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762263.947718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762263/real 1789762263] req@ffff8dd211834000 x1876699496118144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762263.947725:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496118144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762263.947731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762263.947732:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762263.947734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496118144:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762264.523329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762265.547389:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762266.571391:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762267.595361:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762268.619429:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762269.067358:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762269.067370:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762269.067375:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762269.067387:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2248 00000100:00080000:2.0:1789762269.067400:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762269.067410:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762269.067467:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496118272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762269.067525:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762269.067530:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762269.067533:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762269.067538:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2248 00000100:00080000:2.0:1789762269.067543:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762269.067548:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762269.067562:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762269.067585:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496118400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762269.067599:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496118272:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762269.067604:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762269.067614:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762269.067617:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762269.067623:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2248 00000100:00080000:2.0:1789762269.067626:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762269.067631:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762269.067637:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762269.067654:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496118528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762269.067666:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762269.067670:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762269.067672:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762269.067676:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2248 00000100:00080000:2.0:1789762269.067682:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762269.067686:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762269.067692:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762269.067702:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762269.067711:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496118656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762269.067776:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762269/real 1789762269] req@ffff8dd205265500 x1876699496118272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762269.067801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496118272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762269.067815:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762269.067820:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762269.067826:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496118272:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762269.067855:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496118400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762269.067886:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762269.067902:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762269/real 1789762269] req@ffff8dd205266680 x1876699496118400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762269.067916:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496118400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762269.067926:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762269.067930:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762269.067934:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496118400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762269.067941:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496118528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762269.067960:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762269.067971:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762269/real 1789762269] req@ffff8dd205267480 x1876699496118528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762269.068003:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496118528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762269.068032:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762269.068036:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762269.068057:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496118528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762269.068064:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496118656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762269.068088:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762269.068101:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762269/real 1789762269] req@ffff8dd205264a80 x1876699496118656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762269.068133:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496118656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762269.068143:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762269.068147:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762269.068151:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496118656:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762269.643329:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762270.667376:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762271.691366:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762272.715296:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762273.739308:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762274.187301:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762274.187310:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762274.187313:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762274.187322:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2253 00000100:00080000:0.0:1789762274.187332:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762274.187339:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762274.187383:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496118784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762274.187404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762274.187407:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762274.187408:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762274.187411:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2253 00000100:00080000:0.0:1789762274.187413:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762274.187415:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762274.187423:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762274.187431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496118912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762274.187439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762274.187442:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762274.187443:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762274.187448:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2253 00000100:00080000:0.0:1789762274.187450:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762274.187450:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496118784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762274.187457:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762274.187462:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762274.187470:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496119040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762274.187477:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762274.187479:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762274.187480:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762274.187482:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2253 00000100:00080000:0.0:1789762274.187485:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762274.187488:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762274.187491:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762274.187497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496119168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762274.187518:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762274.187557:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762274/real 1789762274] req@ffff8dd211835f80 x1876699496118784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762274.187583:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496118784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762274.187591:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762274.187594:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762274.187597:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496118784:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762274.187617:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496118912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762274.187634:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762274.187643:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762274/real 1789762274] req@ffff8dd211834380 x1876699496118912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762274.187651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496118912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762274.187657:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762274.187660:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762274.187662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496118912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762274.187665:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496119040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762274.187676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762274.187682:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762274/real 1789762274] req@ffff8dd211836680 x1876699496119040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762274.187689:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496119040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762274.187694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762274.187696:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762274.187698:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496119040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762274.187701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496119168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762274.187711:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762274.187717:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762274/real 1789762274] req@ffff8dd211834a80 x1876699496119168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762274.187724:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496119168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762274.187729:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762274.187730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762274.187733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496119168:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762274.763296:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762275.787353:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762276.811290:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762277.835319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762278.859322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762279.307320:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762279.307329:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762279.307332:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762279.307341:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2258 00000100:00080000:2.0:1789762279.307351:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762279.307357:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762279.307391:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496119296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762279.307419:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762279.307423:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762279.307424:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762279.307427:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2258 00000100:00080000:2.0:1789762279.307430:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762279.307441:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762279.307450:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762279.307460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496119424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762279.307469:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762279.307471:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762279.307473:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762279.307475:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2258 00000100:00080000:2.0:1789762279.307477:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762279.307480:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762279.307484:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762279.307487:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496119296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762279.307495:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496119552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762279.307510:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762279.307512:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762279.307514:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762279.307516:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2258 00000100:00080000:2.0:1789762279.307520:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762279.307523:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762279.307526:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762279.307536:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496119680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762279.307591:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496119424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762279.307650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496119552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762279.307669:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496119680:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762279.883398:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762280.907420:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762281.931406:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789762282.379425:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789762282.379465:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789762282.379483:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762282.379526:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762282.379538:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762282.379551:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789762282.379579:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762279/real 1789762282] req@ffff8dd205267800 x1876699496119552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762282.379605:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496119552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762282.379618:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762282.379624:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762282.379629:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496119552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762282.379638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762279/real 1789762282] req@ffff8dd205264380 x1876699496119680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762282.379660:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496119680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762282.379668:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762282.379671:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762282.379675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496119680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762282.379683:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762279/real 1789762282] req@ffff8dd205264e00 x1876699496119296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762282.379697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496119296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762282.379710:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762282.379714:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762282.379718:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496119296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762282.379726:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762279/real 1789762282] req@ffff8dd205267100 x1876699496119424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762282.379740:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496119424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762282.379747:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762282.379751:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762282.379754:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496119424:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762282.955291:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762283.979361:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762284.427301:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762284.427312:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762284.427316:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762284.427328:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2264 00000100:00080000:2.0:1789762284.427341:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762284.427348:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762284.427392:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496119808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762284.427421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762284.427426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762284.427428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762284.427433:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2264 00000100:00080000:2.0:1789762284.427438:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762284.427444:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762284.427456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762284.427475:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496119936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762284.427482:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496119808:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762284.427495:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762284.427501:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762284.427504:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762284.427511:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2264 00000100:00080000:2.0:1789762284.427516:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762284.427520:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762284.427526:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762284.427544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496120064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762284.427554:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762284.427570:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762284.427574:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762284.427576:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762284.427581:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2264 00000100:00080000:2.0:1789762284.427586:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762284.427592:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762284.427598:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762284.427615:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762284/real 1789762284] req@ffff8dd205267b80 x1876699496119808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762284.427615:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496120192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762284.427674:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496119808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762284.427694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762284.427699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762284.427704:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496119808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762284.427732:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496119936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762284.427762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762284.427777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762284/real 1789762284] req@ffff8dd205264000 x1876699496119936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762284.427791:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496119936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762284.427799:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762284.427803:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762284.427806:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496119936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762284.427813:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496120064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762284.427834:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762284.427847:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762284/real 1789762284] req@ffff8dd205265880 x1876699496120064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762284.427860:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496120064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762284.427869:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762284.427873:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762284.427876:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496120064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762284.427883:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496120192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762284.427905:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762284.427918:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762284/real 1789762284] req@ffff8dd205266a00 x1876699496120192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762284.427930:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496120192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762284.427937:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762284.427941:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762284.427944:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496120192:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762285.003336:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762286.027388:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762287.051371:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762288.075327:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762289.099363:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762289.547314:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762289.547331:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762289.547337:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762289.547348:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2269 00000100:00080000:2.0:1789762289.547377:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762289.547385:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762289.547442:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496120320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762289.547475:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762289.547479:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762289.547481:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762289.547484:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2269 00000100:00080000:2.0:1789762289.547506:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762289.547510:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762289.547521:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762289.547537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496120448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762289.547548:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762289.547556:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496120320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762289.547578:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762289.547584:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762289.547590:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2269 00000100:00080000:2.0:1789762289.547592:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762289.547595:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762289.547599:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762289.547609:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496120576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762289.547616:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762289.547619:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762289.547621:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762289.547623:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2269 00000100:00080000:2.0:1789762289.547626:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762289.547628:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762289.547633:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762289.547640:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496120704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762289.547673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762289.547723:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762289/real 1789762289] req@ffff8dd205264700 x1876699496120320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762289.547758:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496120320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762289.547774:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762289.547780:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762289.547783:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496120320:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762289.547821:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496120448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762289.547845:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762289.547857:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762289/real 1789762289] req@ffff8dd205266300 x1876699496120448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762289.547868:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496120448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762289.547886:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762289.547889:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762289.547892:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496120448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762289.547897:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496120576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762289.547912:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762289.547920:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762289/real 1789762289] req@ffff8dd205265180 x1876699496120576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762289.547929:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496120576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762289.547935:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762289.547937:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762289.547940:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496120576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762289.547944:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496120704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762289.547957:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762289.547964:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762289/real 1789762289] req@ffff8dd205266d80 x1876699496120704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762289.547972:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496120704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762289.547979:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762289.547982:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762289.547984:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496120704:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762290.123314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762291.147347:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762292.171295:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762293.195385:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762294.219329:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762294.667271:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762294.667290:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762294.667295:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762294.667322:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2274 00000100:00080000:1.0:1789762294.667335:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762294.667342:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762294.667387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496120832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762294.667425:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762294.667429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762294.667430:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762294.667434:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2274 00000100:00080000:1.0:1789762294.667437:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762294.667441:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762294.667453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762294.667475:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496120832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762294.667488:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496120960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762294.667502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762294.667507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762294.667510:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762294.667515:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2274 00000100:00080000:1.0:1789762294.667520:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762294.667524:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762294.667529:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762294.667539:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496121088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762294.667562:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762294.667567:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789762294.667568:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762294.667571:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762294.667579:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2274 00000100:00080000:1.0:1789762294.667584:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762294.667590:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762294.667596:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762294.667608:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496121216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762294.667617:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762294/real 1789762294] req@ffff8dd20504c000 x1876699496120832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762294.667641:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496120832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762294.667653:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762294.667658:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762294.667662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496120832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762294.667689:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496120960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762294.667714:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762294.667729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762294/real 1789762294] req@ffff8dd20504d880 x1876699496120960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762294.667744:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496120960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762294.667754:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762294.667759:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762294.667763:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496120960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762294.667770:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496121088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762294.667791:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762294.667800:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762294/real 1789762294] req@ffff8dd20504ed80 x1876699496121088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762294.667809:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496121088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762294.667815:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762294.667817:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762294.667819:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496121088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762294.667823:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496121216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762294.667835:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762294.667841:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762294/real 1789762294] req@ffff8dd20504f100 x1876699496121216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762294.667848:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496121216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762294.667854:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762294.667857:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762294.667859:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496121216:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762295.243381:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762296.267327:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762297.291330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762298.315337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762299.339329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762299.787317:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762299.787329:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762299.787334:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762299.787346:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2279 00000100:00080000:3.0:1789762299.787357:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762299.787367:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762299.787405:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496121344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762299.787436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762299.787440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762299.787443:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762299.787447:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2279 00000100:00080000:3.0:1789762299.787451:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762299.787454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762299.787463:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762299.787495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496121472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762299.787501:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496121344:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762299.787508:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762299.787513:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762299.787514:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762299.787518:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2279 00000100:00080000:3.0:1789762299.787522:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762299.787525:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762299.787529:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762299.787540:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496121600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762299.787548:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762299.787551:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762299.787553:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762299.787555:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2279 00000100:00080000:3.0:1789762299.787558:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762299.787562:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762299.787563:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762299.787571:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762299.787589:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496121728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762299.787605:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762299/real 1789762299] req@ffff8dd203147b80 x1876699496121344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762299.787621:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496121344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762299.787629:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762299.787632:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762299.787636:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496121344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762299.787654:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496121472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762299.787671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762299.787690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762299/real 1789762299] req@ffff8dd203146d80 x1876699496121472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762299.787699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496121472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762299.787705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762299.787708:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762299.787711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496121472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762299.787716:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496121600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762299.787729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762299.787736:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762299/real 1789762299] req@ffff8dd203144e00 x1876699496121600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762299.787745:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496121600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762299.787751:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762299.787753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762299.787755:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496121600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762299.787759:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496121728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762299.787770:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762299.787777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762299/real 1789762299] req@ffff8dd203146a00 x1876699496121728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762299.787785:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496121728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762299.787791:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762299.787793:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762299.787795:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496121728:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762300.363316:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762301.387342:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762302.411307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762303.435341:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762304.459340:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762304.907287:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762304.907319:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762304.907323:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762304.907343:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2284 00000100:00080000:3.0:1789762304.907353:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762304.907359:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762304.907413:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496121856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762304.907458:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762304.907461:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762304.907462:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762304.907465:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2284 00000100:00080000:3.0:1789762304.907468:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762304.907471:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762304.907479:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762304.907489:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496121856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762304.907491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496121984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762304.907502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762304.907505:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762304.907506:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762304.907508:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2284 00000100:00080000:3.0:1789762304.907510:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762304.907512:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762304.907515:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762304.907523:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496122112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762304.907531:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762304.907533:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762304.907534:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762304.907536:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2284 00000100:00080000:3.0:1789762304.907540:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762304.907542:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762304.907545:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762304.907553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496122240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762304.907564:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762304.907598:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762304/real 1789762304] req@ffff8dd203145500 x1876699496121856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762304.907622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496121856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762304.907630:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762304.907632:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762304.907636:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496121856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762304.907653:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496121984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762304.907669:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762304.907681:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762304/real 1789762304] req@ffff8dd203144380 x1876699496121984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762304.907694:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496121984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762304.907703:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762304.907706:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762304.907710:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496121984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762304.907717:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496122112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762304.907735:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762304.907749:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762304/real 1789762304] req@ffff8dd203145f80 x1876699496122112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762304.907762:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496122112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762304.907768:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762304.907770:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762304.907772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496122112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762304.907776:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496122240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762304.907786:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762304.907792:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762304/real 1789762304] req@ffff8dd203144700 x1876699496122240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762304.907799:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496122240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762304.907804:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762304.907805:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762304.907807:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496122240:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762305.483310:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762306.507305:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762307.531320:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762308.555494:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762309.579429:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762310.027246:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762310.027255:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762310.027259:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762310.027267:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2289 00000100:00080000:3.0:1789762310.027276:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762310.027282:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762310.027316:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496122368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762310.027339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762310.027342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762310.027343:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762310.027346:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2289 00000100:00080000:3.0:1789762310.027349:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762310.027351:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762310.027370:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762310.027380:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496122496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762310.027389:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762310.027395:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496122368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762310.027395:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762310.027399:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762310.027408:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2289 00000100:00080000:3.0:1789762310.027410:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762310.027413:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762310.027416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762310.027427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496122624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762310.027434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762310.027436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762310.027438:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762310.027441:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2289 00000100:00080000:3.0:1789762310.027444:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762310.027447:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762310.027453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762310.027462:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762310.027469:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496122752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762310.027500:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762310/real 1789762310] req@ffff8dd203144a80 x1876699496122368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762310.027516:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496122368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762310.027526:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762310.027529:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762310.027532:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496122368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762310.027551:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496122496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762310.027566:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762310.027575:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762310/real 1789762310] req@ffff8dd203146300 x1876699496122496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762310.027584:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496122496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762310.027590:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762310.027592:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762310.027594:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496122496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762310.027598:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496122624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762310.027614:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762310.027625:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762310/real 1789762310] req@ffff8dd203147800 x1876699496122624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762310.027638:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496122624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762310.027647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762310.027651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762310.027655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496122624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762310.027662:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496122752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762310.027683:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762310.027692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762310/real 1789762310] req@ffff8dd203145180 x1876699496122752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762310.027701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496122752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762310.027706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762310.027709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762310.027711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496122752:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762310.603345:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762311.627375:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762312.651365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762313.675311:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762314.699394:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762315.147399:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762315.147410:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762315.147416:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762315.147429:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2294 00000100:00080000:2.0:1789762315.147442:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762315.147470:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762315.147538:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496122880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762315.147575:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762315.147578:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762315.147580:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762315.147600:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2294 00000100:00080000:2.0:1789762315.147605:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762315.147610:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762315.147622:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762315.147639:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496122880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762315.147639:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496123008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762315.147675:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762315.147680:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762315.147682:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762315.147689:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2294 00000100:00080000:2.0:1789762315.147694:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762315.147698:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762315.147704:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762315.147735:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496123136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762315.147746:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762315.147760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789762315.147761:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762315.147766:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762315.147773:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2294 00000100:00080000:2.0:1789762315.147777:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762315.147781:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762315.147788:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762315.147802:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496123264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762315.147854:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762315/real 1789762315] req@ffff8dd205265f80 x1876699496122880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762315.147888:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496122880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762315.147899:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762315.147903:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762315.147906:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496122880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762315.147933:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496123008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762315.147955:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762315.147965:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762315/real 1789762315] req@ffff8dd205266d80 x1876699496123008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762315.147975:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496123008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762315.147981:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762315.147984:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762315.147986:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496123008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762315.147990:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496123136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762315.148005:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762315.148011:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762315/real 1789762315] req@ffff8dd205265180 x1876699496123136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762315.148020:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496123136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762315.148025:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762315.148027:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762315.148029:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496123136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762315.148033:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496123264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762315.148050:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762315.148061:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762315/real 1789762315] req@ffff8dd205266300 x1876699496123264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762315.148075:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496123264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762315.148086:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762315.148089:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762315.148094:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496123264:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762315.723436:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762316.747287:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762317.771289:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762318.795296:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762319.819387:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762320.267267:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762320.267276:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762320.267280:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762320.267290:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2299 00000100:00080000:2.0:1789762320.267301:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762320.267308:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762320.267342:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496123392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762320.267367:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762320.267371:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762320.267372:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762320.267376:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2299 00000100:00080000:2.0:1789762320.267378:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762320.267381:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762320.267389:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762320.267400:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496123520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762320.267411:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762320.267415:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762320.267417:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762320.267425:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2299 00000100:00100000:0.0:1789762320.267427:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496123392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762320.267431:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762320.267441:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762320.267448:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762320.267464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496123648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762320.267478:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762320.267482:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762320.267483:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762320.267487:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2299 00000100:00080000:2.0:1789762320.267491:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762320.267495:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762320.267499:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762320.267510:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496123776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762320.267520:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762320.267571:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762320/real 1789762320] req@ffff8dd205264700 x1876699496123392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762320.267603:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496123392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762320.267616:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762320.267621:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762320.267626:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496123392:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762320.267637:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496123520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762320.267663:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762320.267678:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762320/real 1789762320] req@ffff8dd205266a00 x1876699496123520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762320.267692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496123520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762320.267702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762320.267706:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762320.267710:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496123520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762320.267738:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496123648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762320.267758:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762320.267770:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762320/real 1789762320] req@ffff8dd205265880 x1876699496123648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762320.267783:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496123648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762320.267792:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762320.267817:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762320.267822:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496123648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762320.267829:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496123776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762320.267864:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762320.267878:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762320/real 1789762320] req@ffff8dd205264000 x1876699496123776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762320.267893:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496123776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762320.267903:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762320.267907:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762320.267912:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496123776:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762320.843289:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762321.867296:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762322.891363:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762323.915317:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762324.939340:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762325.387262:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762325.387273:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762325.387278:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762325.387289:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2304 00000100:00080000:2.0:1789762325.387300:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762325.387308:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762325.387363:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496123904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762325.387405:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762325.387408:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762325.387409:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762325.387413:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2304 00000100:00080000:2.0:1789762325.387416:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762325.387419:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762325.387427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762325.387438:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496123904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762325.387441:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496124032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762325.387452:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762325.387455:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762325.387456:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762325.387460:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2304 00000100:00080000:2.0:1789762325.387462:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762325.387464:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762325.387468:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762325.387476:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496124160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762325.387483:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762325.387486:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762325.387488:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762325.387490:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2304 00000100:00080000:2.0:1789762325.387492:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762325.387494:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789762325.387516:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789762325.387518:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762325.387529:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496124288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762325.387552:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762325/real 1789762325] req@ffff8dd205267b80 x1876699496123904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762325.387568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496123904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762325.387593:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762325.387596:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762325.387599:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496123904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762325.387618:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496124032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762325.387633:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762325.387641:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762325/real 1789762325] req@ffff8dd205267100 x1876699496124032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762325.387649:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496124032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762325.387654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762325.387656:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762325.387659:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496124032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762325.387662:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496124160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762325.387672:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762325.387702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762325/real 1789762325] req@ffff8dd205264e00 x1876699496124160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762325.387718:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496124160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762325.387723:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762325.387725:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762325.387727:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496124160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762325.387731:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496124288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762325.387744:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762325.387751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762325/real 1789762325] req@ffff8dd205264380 x1876699496124288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762325.387772:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496124288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762325.387778:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762325.387780:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762325.387783:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496124288:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762325.963381:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762326.987333:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762328.011288:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762329.035304:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762330.059353:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762330.507279:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762330.507287:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762330.507290:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762330.507298:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2310 00000100:00080000:2.0:1789762330.507308:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762330.507314:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762330.507346:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496124416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762330.507369:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762330.507373:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762330.507376:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762330.507381:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2310 00000100:00080000:2.0:1789762330.507385:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762330.507389:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762330.507400:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762330.507414:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496124416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762330.507417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496124544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762330.507427:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762330.507430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762330.507431:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762330.507434:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2310 00000100:00080000:2.0:1789762330.507437:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762330.507440:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762330.507445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762330.507457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496124672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762330.507468:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762330.507472:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762330.507476:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762330.507480:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762330.507487:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2310 00000100:00080000:2.0:1789762330.507491:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762330.507496:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762330.507502:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762330.507509:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762330/real 1789762330] req@ffff8dd205267800 x1876699496124416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762330.507515:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496124800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762330.507528:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496124416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762330.507542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762330.507545:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762330.507548:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496124416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762330.507565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496124544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762330.507580:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762330.507588:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762330/real 1789762330] req@ffff8dd205264a80 x1876699496124544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762330.507596:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496124544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762330.507601:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762330.507604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762330.507606:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496124544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762330.507609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496124672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762330.507620:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762330.507626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762330/real 1789762330] req@ffff8dd205267480 x1876699496124672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762330.507633:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496124672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762330.507639:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762330.507640:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762330.507642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496124672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762330.507646:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496124800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762330.507656:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762330.507661:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762330/real 1789762330] req@ffff8dd205266680 x1876699496124800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762330.507668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496124800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762330.507673:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762330.507675:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762330.507677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496124800:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762331.083304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762332.107313:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762333.131442:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762334.155382:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762335.179371:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762335.627291:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762335.627304:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762335.627310:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762335.627325:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2315 00000100:00080000:2.0:1789762335.627339:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762335.627350:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762335.627397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496124928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762335.627432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762335.627437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762335.627439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762335.627444:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2315 00000100:00080000:2.0:1789762335.627448:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762335.627453:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762335.627465:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762335.627483:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496125056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762335.627497:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762335.627503:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762335.627505:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762335.627511:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2315 00000100:00080000:2.0:1789762335.627520:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762335.627522:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496124928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762335.627527:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762335.627542:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762335.627566:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496125184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762335.627589:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762335.627621:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762335.627624:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762335.627629:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2315 00000100:00080000:2.0:1789762335.627636:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762335.627638:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762335.627646:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762335.627656:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762335.627675:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496125312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762335.627717:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762335/real 1789762335] req@ffff8dd205265500 x1876699496124928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762335.627745:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496124928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762335.627759:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762335.627766:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762335.627772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496124928:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762335.627784:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496125056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762335.627816:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762335.627834:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762335/real 1789762335] req@ffff8dd205266680 x1876699496125056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762335.627852:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496125056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762335.627865:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762335.627870:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762335.627874:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496125056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762335.627903:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496125184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762335.627934:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762335.627952:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762335/real 1789762335] req@ffff8dd205267480 x1876699496125184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762335.627972:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496125184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762335.627984:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762335.627989:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762335.627994:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496125184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762335.628002:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496125312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762335.628030:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762335.628065:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762335/real 1789762335] req@ffff8dd205264a80 x1876699496125312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762335.628089:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496125312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762335.628115:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762335.628119:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762335.628123:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496125312:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762336.203335:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762337.227302:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762338.251362:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762339.275321:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762340.299286:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762340.747253:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762340.747261:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762340.747266:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762340.747274:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2320 00000100:00080000:2.0:1789762340.747284:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762340.747290:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762340.747321:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496125440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762340.747344:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762340.747347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762340.747349:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762340.747352:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2320 00000100:00080000:2.0:1789762340.747354:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762340.747356:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762340.747364:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762340.747378:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496125568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762340.747390:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762340.747395:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496125440:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762340.747395:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762340.747400:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762340.747416:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2320 00000100:00080000:2.0:1789762340.747420:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762340.747423:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762340.747428:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762340.747438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496125696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762340.747451:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762340.747453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762340.747454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762340.747459:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2320 00000800:00000100:0.0:1789762340.747461:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762340.747462:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762340.747466:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762340.747469:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762340.747478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496125824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762340.747510:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762340/real 1789762340] req@ffff8dd205267800 x1876699496125440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762340.747530:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496125440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762340.747541:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762340.747546:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762340.747550:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496125440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762340.747574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496125568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762340.747600:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762340.747615:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762340/real 1789762340] req@ffff8dd205264380 x1876699496125568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762340.747628:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496125568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762340.747637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762340.747641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762340.747662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496125568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762340.747669:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496125696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762340.747709:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762340.747721:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762340/real 1789762340] req@ffff8dd205264e00 x1876699496125696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762340.747733:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496125696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762340.747742:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762340.747745:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762340.747748:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496125696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762340.747754:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496125824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762340.747772:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762340.747784:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762340/real 1789762340] req@ffff8dd205267100 x1876699496125824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762340.747796:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496125824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762340.747803:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762340.747807:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762340.747810:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496125824:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762341.323321:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762342.347336:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762343.371316:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762344.395469:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762345.419449:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762345.867354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762345.867367:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762345.867372:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762345.867385:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2325 00000100:00080000:2.0:1789762345.867399:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762345.867408:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762345.867476:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496125952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762345.867536:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762345.867542:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762345.867564:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762345.867573:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2325 00000100:00080000:2.0:1789762345.867579:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762345.867586:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762345.867603:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762345.867682:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496126080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762345.867700:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762345.867705:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762345.867709:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762345.867716:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2325 00000100:00080000:2.0:1789762345.867722:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762345.867731:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762345.867744:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496125952:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789762345.867745:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762345.867774:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496126208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762345.867789:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762345.867792:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762345.867795:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762345.867803:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2325 00000100:00080000:2.0:1789762345.867809:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762345.867814:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762345.867821:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762345.867842:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496126336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762345.867943:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496126080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762345.868016:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496126208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762345.868050:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496126336:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762346.443513:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762347.467333:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762348.491443:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762348.939675:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789762348.939746:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762348.939770:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762348.939827:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762348.939841:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762348.939851:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789762348.939944:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762345/real 1789762348] req@ffff8dd205265880 x1876699496126208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762348.940025:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496126208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762348.940044:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762348.940050:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762348.940057:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496126208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762348.940071:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762345/real 1789762348] req@ffff8dd205266a00 x1876699496126336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762348.940098:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496126336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762348.940114:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762348.940120:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762348.940127:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496126336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762348.940235:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762345/real 1789762348] req@ffff8dd205267b80 x1876699496125952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762348.940265:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496125952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762348.940283:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762348.940301:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762348.940308:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496125952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762348.940321:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762345/real 1789762348] req@ffff8dd205264000 x1876699496126080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762348.940346:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496126080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762348.940371:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762348.940378:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762348.940385:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496126080:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762349.515409:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762350.539416:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762350.987281:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762350.987297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762350.987303:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762350.987330:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2330 00000100:00080000:0.0:1789762350.987346:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762350.987357:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762350.987411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496126464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762350.987439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762350.987443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762350.987444:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762350.987448:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2330 00000100:00080000:0.0:1789762350.987453:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762350.987457:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762350.987481:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762350.987494:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496126464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762350.987503:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496126592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762350.987518:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762350.987523:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762350.987525:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762350.987531:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2330 00000100:00080000:0.0:1789762350.987536:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762350.987542:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762350.987549:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762350.987561:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496126720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762350.987589:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762350.987593:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789762350.987594:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762350.987598:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762350.987603:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2330 00000100:00080000:0.0:1789762350.987607:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762350.987612:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762350.987617:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762350.987633:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496126848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762350.987653:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762350/real 1789762350] req@ffff8dd211836d80 x1876699496126464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762350.987679:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496126464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762350.987694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762350.987700:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762350.987706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496126464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762350.987741:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496126592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762350.987773:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762350.987790:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762350/real 1789762350] req@ffff8dd211835180 x1876699496126592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762350.987805:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496126592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762350.987824:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762350.987828:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762350.987833:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496126592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762350.987840:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496126720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762350.987866:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762350.987881:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762350/real 1789762350] req@ffff8dd211837480 x1876699496126720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762350.987896:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496126720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762350.987907:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762350.987911:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762350.987915:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496126720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762350.987922:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496126848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762350.987947:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762350.987968:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762350/real 1789762350] req@ffff8dd211834e00 x1876699496126848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762350.987982:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496126848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762350.987990:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762350.987994:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762350.987999:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496126848:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762351.563365:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762352.587296:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762353.611315:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762354.635293:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762355.659351:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762356.107325:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762356.107338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762356.107343:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762356.107357:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2335 00000100:00080000:0.0:1789762356.107370:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762356.107378:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762356.107426:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496126976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762356.107459:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762356.107464:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762356.107466:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762356.107471:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2335 00000100:00080000:0.0:1789762356.107477:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762356.107493:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762356.107507:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762356.107539:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496127104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762356.107543:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496126976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762356.107553:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762356.107561:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762356.107563:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762356.107567:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2335 00000100:00080000:0.0:1789762356.107570:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762356.107574:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762356.107579:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762356.107589:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496127232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762356.107596:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762356.107599:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762356.107601:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762356.107604:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2335 00000100:00080000:0.0:1789762356.107623:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762356.107628:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762356.107633:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762356.107637:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762356.107653:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496127360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762356.107691:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762356/real 1789762356] req@ffff8dd211834e00 x1876699496126976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762356.107712:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496126976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762356.107723:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762356.107728:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762356.107732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496126976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762356.107756:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496127104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762356.107777:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762356.107786:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762356/real 1789762356] req@ffff8dd211837480 x1876699496127104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762356.107794:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496127104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762356.107800:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762356.107802:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762356.107805:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496127104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762356.107809:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496127232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762356.107820:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762356.107830:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762356/real 1789762356] req@ffff8dd211835180 x1876699496127232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762356.107843:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496127232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762356.107853:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762356.107856:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762356.107860:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496127232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762356.107867:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496127360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762356.107890:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762356.107905:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762356/real 1789762356] req@ffff8dd211836d80 x1876699496127360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762356.107920:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496127360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762356.107929:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762356.107933:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762356.107937:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496127360:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762356.683375:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762357.707295:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762358.731378:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762359.755384:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762360.779400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762361.227256:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762361.227264:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762361.227269:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762361.227288:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2340 00000100:00080000:0.0:1789762361.227298:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762361.227305:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762361.227350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496127488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762361.227379:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762361.227383:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762361.227384:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762361.227388:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2340 00000100:00080000:0.0:1789762361.227390:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762361.227393:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762361.227402:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762361.227412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496127616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762361.227421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762361.227439:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762361.227441:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762361.227445:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2340 00000100:00080000:0.0:1789762361.227448:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762361.227452:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762361.227458:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762361.227465:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496127488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762361.227473:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496127744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762361.227525:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762361.227529:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762361.227530:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762361.227534:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2340 00000100:00080000:0.0:1789762361.227538:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762361.227559:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762361.227564:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762361.227576:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496127872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762361.227597:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762361.227647:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762361/real 1789762361] req@ffff8dd211834a80 x1876699496127488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762361.227665:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496127488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762361.227674:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762361.227678:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762361.227681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496127488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762361.227688:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496127616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762361.227710:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762361.227720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762361/real 1789762361] req@ffff8dd211836680 x1876699496127616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762361.227730:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496127616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762361.227737:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762361.227740:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762361.227742:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496127616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762361.227763:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496127744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762361.227777:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762361.227785:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762361/real 1789762361] req@ffff8dd211834380 x1876699496127744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762361.227814:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496127744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762361.227826:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762361.227831:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762361.227835:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496127744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762361.227844:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496127872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762361.227871:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762361.227887:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762361/real 1789762361] req@ffff8dd211835f80 x1876699496127872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762361.227899:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496127872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762361.227907:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762361.227910:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762361.227913:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496127872:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762361.803354:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762362.827280:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762363.851405:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762364.875357:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762365.899337:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762366.347273:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762366.347284:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762366.347288:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762366.347298:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2345 00000100:00080000:0.0:1789762366.347325:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762366.347345:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762366.347402:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496128000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762366.347503:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762366.347509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762366.347512:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762366.347520:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2345 00000100:00080000:0.0:1789762366.347526:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762366.347532:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762366.347556:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496128000:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789762366.347560:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762366.347585:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496128128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762366.347598:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762366.347602:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762366.347604:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762366.347608:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2345 00000100:00080000:0.0:1789762366.347620:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762366.347624:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789762366.347632:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789762366.347634:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762366.347647:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496128256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762366.347658:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762366.347662:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762366.347663:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762366.347666:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2345 00000100:00080000:0.0:1789762366.347669:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762366.347673:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762366.347678:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762366.347689:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762366/real 1789762366] req@ffff8dd211834000 x1876699496128000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762366.347690:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496128384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762366.347738:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496128000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762366.347749:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762366.347753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762366.347757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496128000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762366.347793:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496128128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762366.347817:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762366.347833:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762366/real 1789762366] req@ffff8dd211836a00 x1876699496128128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762366.347851:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496128128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762366.347862:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762366.347866:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762366.347871:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496128128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762366.347879:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496128256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762366.347909:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762366.347927:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762366/real 1789762366] req@ffff8dd211835c00 x1876699496128256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762366.347944:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496128256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762366.347956:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762366.347960:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762366.347965:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496128256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762366.347983:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496128384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762366.348003:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762366.348013:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762366/real 1789762366] req@ffff8dd211834700 x1876699496128384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762366.348022:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496128384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762366.348045:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762366.348048:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762366.348051:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496128384:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762366.923390:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762367.947355:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762368.971334:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762369.995304:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762371.019393:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762371.467404:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762371.467438:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762371.467445:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762371.467456:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2351 00000100:00080000:1.0:1789762371.467470:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762371.467478:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762371.467521:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496128512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762371.467556:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762371.467561:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762371.467563:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762371.467568:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2351 00000100:00080000:1.0:1789762371.467571:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762371.467576:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762371.467587:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762371.467601:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496128640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762371.467613:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496128512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762371.467619:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762371.467634:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762371.467637:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762371.467643:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2351 00000100:00080000:1.0:1789762371.467646:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762371.467651:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762371.467657:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762371.467674:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496128768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762371.467685:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762371.467688:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762371.467690:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762371.467694:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2351 00000100:00080000:1.0:1789762371.467698:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762371.467703:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789762371.467712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789762371.467717:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762371.467744:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496128896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762371.467765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762371/real 1789762371] req@ffff8dd20504ce00 x1876699496128512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762371.467807:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496128512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762371.467821:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762371.467826:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762371.467831:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496128512:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762371.467857:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496128640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762371.467888:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762371.467910:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762371/real 1789762371] req@ffff8dd20504f800 x1876699496128640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762371.467933:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496128640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762371.467949:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762371.467956:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762371.467962:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496128640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762371.467974:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496128768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762371.468004:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762371.468018:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762371/real 1789762371] req@ffff8dd20504fb80 x1876699496128768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762371.468038:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496128768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762371.468048:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762371.468051:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762371.468054:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496128768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762371.468063:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496128896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762371.468092:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762371.468110:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762371/real 1789762371] req@ffff8dd20504df80 x1876699496128896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762371.468132:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496128896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762371.468146:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762371.468151:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762371.468157:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496128896:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762372.043433:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762373.067336:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762374.091359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762375.115373:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762376.139319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762376.587277:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762376.587306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762376.587312:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762376.587324:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2356 00000100:00080000:1.0:1789762376.587338:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762376.587347:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762376.587414:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496129024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762376.587474:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762376.587478:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762376.587481:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762376.587485:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2356 00000100:00080000:1.0:1789762376.587489:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762376.587495:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762376.587505:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762376.587517:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496129024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762376.587523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496129152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762376.587536:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762376.587540:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762376.587542:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762376.587546:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2356 00000100:00080000:1.0:1789762376.587551:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762376.587556:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762376.587579:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762376.587596:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496129280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762376.587599:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762376.587619:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762376.587624:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762376.587627:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762376.587643:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2356 00000100:00080000:1.0:1789762376.587650:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762376.587660:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762376/real 1789762376] req@ffff8dd20504c700 x1876699496129024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762376.587661:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762376.587672:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762376.587699:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496129408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762376.587721:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496129024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762376.587740:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762376.587745:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762376.587751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496129024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762376.587773:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496129152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762376.587792:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762376.587821:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762376/real 1789762376] req@ffff8dd20504d180 x1876699496129152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762376.587833:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496129152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762376.587841:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762376.587844:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762376.587847:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496129152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762376.587853:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496129280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762376.587870:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762376.587878:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762376/real 1789762376] req@ffff8dd20504e300 x1876699496129280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762376.587889:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496129280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762376.587896:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762376.587899:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762376.587902:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496129280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762376.587907:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496129408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762376.587921:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762376.587929:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762376/real 1789762376] req@ffff8dd20504d500 x1876699496129408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762376.587939:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496129408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762376.587946:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762376.587949:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762376.587951:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496129408:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762377.163366:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762378.187448:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762379.211403:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762380.235420:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762381.259324:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762381.707286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762381.707299:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762381.707303:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762381.707311:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2361 00000100:00080000:2.0:1789762381.707320:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762381.707326:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762381.707374:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496129536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762381.707397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762381.707400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762381.707402:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762381.707407:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2361 00000100:00080000:2.0:1789762381.707411:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762381.707417:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762381.707427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762381.707439:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496129536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762381.707444:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496129664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762381.707454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762381.707457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762381.707458:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762381.707461:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2361 00000100:00080000:2.0:1789762381.707464:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762381.707468:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762381.707473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762381.707481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496129792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762381.707489:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762381.707491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762381.707492:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762381.707497:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2361 00000800:00000100:0.0:1789762381.707498:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762381.707501:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762381.707506:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762381.707510:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762381.707519:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496129920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762381.707539:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762381/real 1789762381] req@ffff8dd205264700 x1876699496129536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762381.707553:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496129536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762381.707561:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762381.707563:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762381.707576:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496129536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762381.707600:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496129664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762381.707627:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762381.707637:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762381/real 1789762381] req@ffff8dd205266300 x1876699496129664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762381.707646:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496129664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762381.707652:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762381.707654:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762381.707656:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496129664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762381.707660:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496129792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762381.707673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762381.707679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762381/real 1789762381] req@ffff8dd205265180 x1876699496129792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762381.707686:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496129792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762381.707691:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762381.707693:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762381.707695:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496129792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762381.707699:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496129920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762381.707708:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762381.707714:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762381/real 1789762381] req@ffff8dd205266d80 x1876699496129920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762381.707726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496129920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762381.707731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762381.707733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762381.707734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496129920:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762382.283380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762383.307314:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762384.331364:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762385.355352:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762386.379377:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762386.827249:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762386.827259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762386.827264:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762386.827292:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2366 00000100:00080000:2.0:1789762386.827324:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762386.827350:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762386.827386:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496130048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762386.827452:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762386.827457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762386.827458:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762386.827463:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2366 00000100:00080000:2.0:1789762386.827483:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762386.827487:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762386.827495:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762386.827498:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496130048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762386.827512:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496130176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762386.827526:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762386.827530:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762386.827533:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762386.827537:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2366 00000100:00080000:2.0:1789762386.827542:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762386.827547:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762386.827553:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762386.827554:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762386.827568:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496130304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762386.827576:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762386.827579:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762386.827580:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762386.827584:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2366 00000100:00080000:2.0:1789762386.827588:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762386.827602:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762386/real 1789762386] req@ffff8dd205265f80 x1876699496130048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762386.827602:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762386.827608:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762386.827622:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496130432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762386.827626:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496130048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762386.827644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762386.827647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762386.827651:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496130048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762386.827669:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496130176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762386.827685:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762386.827693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762386/real 1789762386] req@ffff8dd205266d80 x1876699496130176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762386.827701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496130176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762386.827711:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762386.827714:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762386.827716:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496130176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762386.827719:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496130304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762386.827730:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762386.827735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762386/real 1789762386] req@ffff8dd205265180 x1876699496130304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762386.827742:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496130304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762386.827748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762386.827750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762386.827751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496130304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762386.827755:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496130432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762386.827765:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762386.827771:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762386/real 1789762386] req@ffff8dd205266300 x1876699496130432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762386.827786:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496130432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762386.827794:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762386.827797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762386.827801:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496130432:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762387.403337:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762388.427394:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762389.451318:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762390.475380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762391.499343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762391.947242:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762391.947251:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762391.947255:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762391.947276:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2371 00000100:00080000:2.0:1789762391.947302:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762391.947308:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762391.947341:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496130560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762391.947363:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762391.947365:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762391.947367:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762391.947370:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2371 00000100:00080000:2.0:1789762391.947372:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762391.947374:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762391.947382:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762391.947391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496130688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762391.947400:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762391.947402:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496130560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762391.947403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762391.947408:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762391.947412:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2371 00000100:00080000:2.0:1789762391.947414:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762391.947416:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762391.947420:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762391.947429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496130816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762391.947436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762391.947439:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762391.947440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762391.947442:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2371 00000100:00080000:2.0:1789762391.947445:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762391.947448:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762391.947451:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762391.947456:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496130944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762391.947471:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762391.947511:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762391/real 1789762391] req@ffff8dd205264700 x1876699496130560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762391.947524:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496130560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762391.947532:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762391.947534:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762391.947537:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496130560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762391.947568:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496130688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762391.947586:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762391.947595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762391/real 1789762391] req@ffff8dd205264000 x1876699496130688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762391.947603:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496130688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762391.947608:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762391.947611:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762391.947613:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496130688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762391.947616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496130816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762391.947629:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762391.947635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762391/real 1789762391] req@ffff8dd205267b80 x1876699496130816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762391.947643:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496130816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762391.947648:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762391.947650:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762391.947652:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496130816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762391.947655:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496130944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762391.947666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762391.947672:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762391/real 1789762391] req@ffff8dd205266a00 x1876699496130944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762391.947679:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496130944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762391.947684:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762391.947686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762391.947687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496130944:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762392.523331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762393.547322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762394.571358:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762395.595383:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762396.619385:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762397.067293:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762397.067304:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762397.067308:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762397.067318:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2376 00000100:00080000:2.0:1789762397.067330:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762397.067338:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762397.067389:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496131072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762397.067417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762397.067422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762397.067424:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762397.067428:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2376 00000100:00080000:2.0:1789762397.067433:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762397.067438:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762397.067447:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762397.067457:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496131072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762397.067465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496131200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762397.067494:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762397.067509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762397.067511:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762397.067516:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2376 00000100:00080000:2.0:1789762397.067520:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762397.067525:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762397.067532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762397.067544:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496131328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762397.067554:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762397.067563:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762397.067568:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762397.067571:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762397.067577:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2376 00000100:00080000:2.0:1789762397.067581:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762397.067586:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762397.067592:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762397.067604:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496131456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762397.067612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762397/real 1789762397] req@ffff8dd205265880 x1876699496131072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762397.067660:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496131072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762397.067671:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762397.067676:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762397.067679:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496131072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762397.067699:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496131200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762397.067718:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762397.067744:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762397/real 1789762397] req@ffff8dd205267100 x1876699496131200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762397.067753:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496131200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762397.067759:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762397.067761:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762397.067764:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496131200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762397.067768:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496131328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762397.067780:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762397.067787:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762397/real 1789762397] req@ffff8dd205264e00 x1876699496131328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762397.067794:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496131328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762397.067804:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762397.067806:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762397.067808:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496131328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762397.067812:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496131456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762397.067838:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762397.067844:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762397/real 1789762397] req@ffff8dd205264380 x1876699496131456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762397.067851:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496131456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762397.067857:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762397.067858:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762397.067860:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496131456:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762397.643354:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762398.667304:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762399.691431:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762400.715359:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762401.739322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762402.187268:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762402.187290:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762402.187294:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762402.187303:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2381 00000100:00080000:2.0:1789762402.187314:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762402.187320:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762402.187356:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496131584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762402.187382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762402.187385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762402.187387:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762402.187391:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2381 00000100:00080000:2.0:1789762402.187394:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762402.187396:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762402.187405:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762402.187416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496131712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762402.187426:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762402.187431:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496131584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762402.187432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762402.187436:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762402.187444:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2381 00000100:00080000:2.0:1789762402.187446:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762402.187449:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762402.187454:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762402.187463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496131840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762402.187472:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762402.187474:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762402.187476:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762402.187479:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2381 00000100:00080000:2.0:1789762402.187482:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762402.187485:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762402.187489:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762402.187496:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762402.187505:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496131968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762402.187536:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762402/real 1789762402] req@ffff8dd205267800 x1876699496131584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762402.187578:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496131584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762402.187594:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762402.187601:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762402.187619:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496131584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762402.187672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496131712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762402.187696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762402.187723:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762402/real 1789762402] req@ffff8dd205264a80 x1876699496131712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762402.187735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496131712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762402.187747:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762402.187751:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762402.187756:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496131712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762402.187764:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496131840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762402.187793:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762402.187808:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762402/real 1789762402] req@ffff8dd205267480 x1876699496131840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762402.187846:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496131840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762402.187853:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762402.187856:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762402.187858:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496131840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762402.187863:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496131968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762402.187880:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762402.187888:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762402/real 1789762402] req@ffff8dd205266680 x1876699496131968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762402.187897:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496131968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762402.187904:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762402.187906:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762402.187908:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496131968:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762402.763438:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762403.787298:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762404.811276:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762405.835283:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762406.859360:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762407.307280:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762407.307290:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762407.307295:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762407.307304:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2386 00000100:00080000:1.0:1789762407.307315:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762407.307322:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762407.307359:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496132096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762407.307405:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762407.307410:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762407.307412:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762407.307417:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2386 00000100:00080000:1.0:1789762407.307432:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762407.307437:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762407.307449:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762407.307465:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496132096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762407.307468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496132224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762407.307529:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762407.307557:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762407.307562:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762407.307571:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2386 00000100:00080000:1.0:1789762407.307591:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762407.307601:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762407.307609:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762407.307632:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762407.307677:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496132352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762407.307687:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762407/real 1789762407] req@ffff8dd20504e680 x1876699496132096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762407.307716:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762407.307727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496132096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762407.307728:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762407.307737:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762407.307753:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2386 00000100:00080000:0.0:1789762407.307754:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762407.307760:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762407.307765:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1789762407.307772:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762407.307777:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496132096:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789762407.307796:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762407.307812:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496132224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762407.307822:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496132480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762407.307853:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762407.307889:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762407/real 1789762407] req@ffff8dd20504d500 x1876699496132224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762407.307902:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496132224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762407.307916:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762407.307919:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762407.307922:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496132224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762407.307928:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496132352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762407.307944:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762407.307953:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762407/real 1789762407] req@ffff8dd20504e300 x1876699496132352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762407.307985:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496132352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762407.307995:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762407.307999:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762407.308003:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496132352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762407.308021:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496132480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762407.308042:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762407.308054:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762407/real 1789762407] req@ffff8dd20504d180 x1876699496132480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762407.308068:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496132480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762407.308097:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762407.308103:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762407.308110:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496132480:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762407.883294:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762408.907338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762409.931310:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762410.955314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762411.979328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762412.427239:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762412.427246:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762412.427249:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762412.427257:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2392 00000100:00080000:1.0:1789762412.427267:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762412.427272:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762412.427303:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496132608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762412.427325:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762412.427327:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762412.427329:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762412.427332:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2392 00000100:00080000:1.0:1789762412.427334:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762412.427336:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762412.427343:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762412.427353:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496132736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762412.427361:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762412.427363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762412.427365:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762412.427369:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2392 00000100:00100000:0.0:1789762412.427370:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496132608:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762412.427371:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762412.427378:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762412.427384:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762412.427398:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496132864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762412.427410:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762412.427413:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762412.427415:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762412.427420:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2392 00000100:00080000:1.0:1789762412.427425:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762412.427429:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762412.427436:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762412.427450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496132992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762412.427464:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496132736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762412.427492:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496132864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762412.427507:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496132992:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762413.003278:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762414.027300:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762415.051302:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762415.499401:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789762415.499436:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762415.499449:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762415.499488:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762415.499494:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762415.499500:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789762415.499537:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762412/real 1789762415] req@ffff8dd20504df80 x1876699496132736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762415.499562:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496132736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762415.499573:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762415.499579:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762415.499583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496132736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762415.499591:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762412/real 1789762415] req@ffff8dd20504fb80 x1876699496132864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762415.499603:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496132864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762415.499612:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762415.499615:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762415.499619:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496132864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762415.499628:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762412/real 1789762415] req@ffff8dd20504f800 x1876699496132992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762415.499640:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496132992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762415.499648:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762415.499652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762415.499655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496132992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762415.499662:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762412/real 1789762415] req@ffff8dd20504c700 x1876699496132608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762415.499675:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496132608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762415.499684:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762415.499687:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762415.499691:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496132608:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789762416.075346:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762417.099327:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762417.547270:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762417.547278:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762417.547281:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762417.547291:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2397 00000100:00080000:1.0:1789762417.547302:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762417.547309:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762417.547340:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496133120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762417.547368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762417.547372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762417.547374:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762417.547378:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2397 00000100:00080000:1.0:1789762417.547382:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762417.547387:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762417.547397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762417.547415:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496133248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762417.547421:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496133120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762417.547441:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762417.547446:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762417.547448:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762417.547454:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2397 00000100:00080000:1.0:1789762417.547460:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762417.547465:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762417.547471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762417.547487:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496133376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762417.547498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762417.547504:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762417.547506:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762417.547510:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2397 00000100:00080000:1.0:1789762417.547515:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762417.547525:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789762417.547526:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789762417.547536:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762417.547552:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496133504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762417.547591:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762417/real 1789762417] req@ffff8dd20504ce00 x1876699496133120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762417.547616:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496133120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762417.547629:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762417.547636:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762417.547640:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496133120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762417.547680:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496133248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762417.547710:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762417.547726:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762417/real 1789762417] req@ffff8dd20504f100 x1876699496133248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762417.547739:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496133248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762417.547749:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762417.547753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762417.547757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496133248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762417.547765:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496133376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762417.547789:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762417.547802:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762417/real 1789762417] req@ffff8dd20504ed80 x1876699496133376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762417.547815:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496133376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762417.547824:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762417.547828:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762417.547831:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496133376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762417.547838:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496133504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762417.547860:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762417.547872:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762417/real 1789762417] req@ffff8dd20504d880 x1876699496133504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762417.547886:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496133504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762417.547894:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762417.547898:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762417.547901:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496133504:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762418.123306:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762419.147317:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762420.171322:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762421.195347:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762422.219329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762422.667292:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762422.667303:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762422.667308:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762422.667319:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2402 00000100:00080000:3.0:1789762422.667331:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762422.667339:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762422.667387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496133632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762422.667414:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762422.667418:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762422.667420:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762422.667425:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2402 00000100:00080000:3.0:1789762422.667428:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762422.667431:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762422.667441:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762422.667454:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496133760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762422.667468:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762422.667474:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496133632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762422.667475:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762422.667480:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762422.667492:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2402 00000100:00080000:3.0:1789762422.667496:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762422.667500:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762422.667505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762422.667527:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496133888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762422.667538:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762422.667542:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762422.667544:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762422.667547:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2402 00000100:00080000:3.0:1789762422.667551:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762422.667555:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762422.667560:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762422.667573:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496134016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762422.667588:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762422.667658:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762422/real 1789762422] req@ffff8dd203145c00 x1876699496133632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762422.667690:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496133632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762422.667704:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762422.667709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762422.667714:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496133632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762422.667739:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496133760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762422.667766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762422.667780:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762422/real 1789762422] req@ffff8dd203147480 x1876699496133760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762422.667801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496133760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762422.667810:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762422.667813:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762422.667817:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496133760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762422.667823:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496133888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762422.667840:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762422.667850:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762422/real 1789762422] req@ffff8dd203144000 x1876699496133888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762422.667861:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496133888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762422.667870:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762422.667873:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762422.667885:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496133888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762422.667891:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496134016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762422.667907:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762422.667916:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762422/real 1789762422] req@ffff8dd203147100 x1876699496134016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762422.667927:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496134016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762422.667971:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762422.667975:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762422.667980:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496134016:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762423.243442:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762424.267404:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762425.291408:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762426.315307:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762427.339365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762427.787339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762427.787352:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762427.787358:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762427.787372:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2407 00000100:00080000:2.0:1789762427.787404:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762427.787413:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762427.787465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496134144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762427.787498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762427.787502:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762427.787504:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762427.787508:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2407 00000100:00080000:2.0:1789762427.787511:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762427.787515:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762427.787527:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762427.787546:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496134272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762427.787548:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496134144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762427.787568:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762427.787571:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762427.787573:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762427.787577:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2407 00000100:00080000:2.0:1789762427.787582:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762427.787585:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762427.787591:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762427.787603:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496134400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762427.787614:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762427.787618:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762427.787620:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762427.787623:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2407 00000100:00080000:2.0:1789762427.787627:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762427.787632:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762427.787636:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762427.787646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762427.787650:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496134528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762427.787702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762427/real 1789762427] req@ffff8dd205265500 x1876699496134144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762427.787723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496134144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762427.787734:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762427.787738:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762427.787742:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496134144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762427.787769:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496134272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762427.787790:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762427.787801:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762427/real 1789762427] req@ffff8dd205266680 x1876699496134272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762427.787812:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496134272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762427.787819:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762427.787822:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762427.787826:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496134272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762427.787849:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496134400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762427.787869:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762427.787878:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762427/real 1789762427] req@ffff8dd205267480 x1876699496134400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762427.787888:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496134400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762427.787897:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762427.787899:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762427.787902:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496134400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762427.787907:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496134528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762427.787944:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762427.787956:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762427/real 1789762427] req@ffff8dd205264a80 x1876699496134528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762427.787970:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496134528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762427.788016:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762427.788021:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762427.788026:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496134528:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762428.363355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762429.387394:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762430.411297:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762431.435421:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762432.459388:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762432.907346:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762432.907358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762432.907363:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762432.907373:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2412 00000100:00080000:1.0:1789762432.907383:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762432.907390:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762432.907425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496134656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762432.907467:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762432.907470:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762432.907471:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762432.907475:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2412 00000100:00080000:1.0:1789762432.907477:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762432.907479:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762432.907487:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762432.907496:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496134784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762432.907505:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762432.907507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762432.907510:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762432.907512:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496134656:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762432.907515:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2412 00000100:00080000:1.0:1789762432.907522:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762432.907525:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762432.907528:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762432.907536:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496134912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762432.907547:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762432.907550:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762432.907552:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762432.907554:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2412 00000100:00080000:1.0:1789762432.907566:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762432.907577:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762432.907580:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762432.907585:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762432.907590:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496135040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762432.907627:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762432/real 1789762432] req@ffff8dd20504c000 x1876699496134656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762432.907642:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496134656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762432.907651:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762432.907654:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762432.907657:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496134656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762432.907675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496134784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762432.907690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762432.907698:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762432/real 1789762432] req@ffff8dd20504ea00 x1876699496134784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762432.907706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496134784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762432.907711:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762432.907714:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762432.907716:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496134784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762432.907719:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496134912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762432.907730:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762432.907736:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762432/real 1789762432] req@ffff8dd20504c380 x1876699496134912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762432.907743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496134912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762432.907753:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762432.907755:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762432.907757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496134912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762432.907760:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496135040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762432.907770:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762432.907775:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762432/real 1789762432] req@ffff8dd20504dc00 x1876699496135040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762432.907790:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496135040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762432.907795:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762432.907797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762432.907799:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496135040:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762433.483491:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762434.507391:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762435.531371:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762436.555437:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762437.579419:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762438.027349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762438.027365:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762438.027372:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762438.027388:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2417 00000100:00080000:3.0:1789762438.027412:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762438.027421:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762438.027477:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496135168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762438.027510:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762438.027513:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762438.027514:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762438.027518:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2417 00000100:00080000:3.0:1789762438.027520:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762438.027523:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762438.027534:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762438.027548:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496135296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762438.027558:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762438.027562:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762438.027566:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762438.027567:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496135168:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762438.027571:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2417 00000100:00080000:3.0:1789762438.027577:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762438.027581:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762438.027585:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762438.027594:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496135424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762438.027603:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762438.027606:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762438.027607:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762438.027610:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2417 00000100:00080000:3.0:1789762438.027614:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762438.027617:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762438.027621:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762438.027630:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496135552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762438.027661:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762438.027718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762438/real 1789762438] req@ffff8dd203145880 x1876699496135168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762438.027743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496135168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762438.027757:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762438.027762:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762438.027767:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496135168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762438.027799:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496135296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762438.027839:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762438.027857:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762438/real 1789762438] req@ffff8dd203147100 x1876699496135296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762438.027874:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496135296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762438.027884:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762438.027889:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762438.027893:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496135296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762438.027901:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496135424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762438.027945:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762438.027970:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762438/real 1789762438] req@ffff8dd203144000 x1876699496135424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762438.027985:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496135424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762438.027995:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762438.027999:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762438.028003:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496135424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762438.028011:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496135552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762438.028034:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762438.028048:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762438/real 1789762438] req@ffff8dd203147480 x1876699496135552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762438.028062:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496135552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762438.028073:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762438.028077:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762438.028081:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496135552:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762438.603411:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762439.627352:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762440.651401:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762441.675347:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762442.699300:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762443.147286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762443.147294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762443.147298:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762443.147307:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2422 00000100:00080000:1.0:1789762443.147316:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762443.147334:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762443.147366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496135680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762443.147387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762443.147390:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762443.147391:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762443.147395:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2422 00000100:00080000:1.0:1789762443.147398:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762443.147400:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762443.147408:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762443.147417:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496135808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762443.147425:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762443.147427:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762443.147428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762443.147431:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2422 00000100:00080000:1.0:1789762443.147433:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762443.147435:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762443.147439:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762443.147445:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496135680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762443.147449:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496135936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762443.147460:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762443.147463:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762443.147464:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762443.147476:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2422 00000100:00080000:1.0:1789762443.147478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762443.147481:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762443.147483:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762443.147490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496136064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762443.147525:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762443.147577:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762443/real 1789762443] req@ffff8dd20504ca80 x1876699496135680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762443.147593:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496135680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762443.147601:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762443.147604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762443.147607:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496135680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762443.147612:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496135808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762443.147628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762443.147654:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762443/real 1789762443] req@ffff8dd20504dc00 x1876699496135808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762443.147662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496135808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762443.147672:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762443.147674:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762443.147677:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496135808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762443.147695:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496135936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762443.147718:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762443.147724:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762443/real 1789762443] req@ffff8dd20504c380 x1876699496135936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762443.147732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496135936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762443.147737:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762443.147739:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762443.147740:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496135936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762443.147744:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496136064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762443.147754:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762443.147759:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762443/real 1789762443] req@ffff8dd20504ea00 x1876699496136064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762443.147766:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496136064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762443.147771:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762443.147773:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762443.147775:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496136064:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762443.723318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762444.747383:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762445.771443:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762446.795414:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762447.819336:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762448.267329:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762448.267338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762448.267342:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762448.267352:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2427 00000100:00080000:2.0:1789762448.267362:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762448.267368:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762448.267411:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496136192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762448.267442:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762448.267445:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762448.267447:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762448.267450:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2427 00000100:00080000:2.0:1789762448.267453:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762448.267455:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762448.267464:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762448.267478:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496136320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762448.267493:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496136192:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762448.267493:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762448.267505:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762448.267508:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762448.267514:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2427 00000100:00080000:2.0:1789762448.267519:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762448.267525:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762448.267532:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762448.267548:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496136448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762448.267558:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762448.267561:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762448.267563:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762448.267567:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2427 00000800:00000100:0.0:1789762448.267572:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762448.267578:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762448.267587:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762448.267594:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762448.267610:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496136576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762448.267616:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762448/real 1789762448] req@ffff8dd205267800 x1876699496136192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762448.267652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496136192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762448.267661:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762448.267665:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762448.267668:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496136192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762448.267689:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496136320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762448.267717:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762448.267727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762448/real 1789762448] req@ffff8dd205264380 x1876699496136320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762448.267737:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496136320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762448.267743:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762448.267746:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762448.267749:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496136320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762448.267753:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496136448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762448.267765:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762448.267773:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762448/real 1789762448] req@ffff8dd205264e00 x1876699496136448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762448.267782:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496136448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762448.267788:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762448.267790:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762448.267792:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496136448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762448.267796:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496136576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762448.267808:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762448.267815:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762448/real 1789762448] req@ffff8dd205267100 x1876699496136576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762448.267824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496136576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762448.267830:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762448.267832:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762448.267834:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496136576:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762448.843419:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762449.867281:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762450.891391:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762451.915468:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762452.939395:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762453.387344:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762453.387374:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762453.387381:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762453.387398:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2432 00000100:00080000:2.0:1789762453.387416:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762453.387427:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762453.387502:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496136704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762453.387544:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762453.387550:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762453.387553:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762453.387558:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2432 00000100:00080000:2.0:1789762453.387562:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762453.387567:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762453.387580:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762453.387599:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496136832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762453.387623:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762453.387630:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762453.387634:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762453.387643:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2432 00000100:00080000:2.0:1789762453.387652:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762453.387671:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762453.387684:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762453.387694:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496136704:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762453.387717:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496136960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762453.387751:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762453.387756:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762453.387759:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762453.387766:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2432 00000100:00080000:2.0:1789762453.387773:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762453.387779:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762453.387789:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762453.387832:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496137088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762453.387852:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762453.387945:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762453/real 1789762453] req@ffff8dd205265880 x1876699496136704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762453.387985:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496136704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762453.388000:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762453.388005:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762453.388010:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496136704:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762453.388020:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496136832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762453.388070:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762453.388086:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762453/real 1789762453] req@ffff8dd205266a00 x1876699496136832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762453.388101:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496136832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762453.388112:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762453.388116:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762453.388120:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496136832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762453.388169:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496136960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762453.388191:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762453.388202:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762453/real 1789762453] req@ffff8dd205267b80 x1876699496136960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762453.388214:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496136960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762453.388223:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762453.388286:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762453.388292:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496136960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762453.388302:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496137088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762453.388354:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762453.388369:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762453/real 1789762453] req@ffff8dd205264000 x1876699496137088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762453.388383:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496137088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762453.388393:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762453.388396:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762453.388400:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496137088:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762453.963396:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762454.987461:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762456.011352:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762457.035308:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762458.059302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762458.507282:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762458.507291:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762458.507295:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762458.507304:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2438 00000100:00080000:3.0:1789762458.507316:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762458.507325:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762458.507382:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496137216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762458.507412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762458.507415:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762458.507417:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762458.507421:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2438 00000100:00080000:3.0:1789762458.507425:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762458.507429:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762458.507439:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762458.507459:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496137344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762458.507475:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496137216:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762458.507497:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762458.507503:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762458.507506:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762458.507514:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2438 00000100:00080000:3.0:1789762458.507520:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762458.507525:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762458.507541:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762458.507555:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496137472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762458.507560:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762458.507573:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762458.507576:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762458.507579:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762458.507584:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2438 00000100:00080000:3.0:1789762458.507587:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762458.507590:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762458.507595:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762458.507607:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496137600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762458.507611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762458/real 1789762458] req@ffff8dd203145c00 x1876699496137216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762458.507643:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496137216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762458.507653:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762458.507657:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762458.507660:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496137216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762458.507693:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496137344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762458.507712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762458.507722:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762458/real 1789762458] req@ffff8dd203145180 x1876699496137344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762458.507732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496137344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762458.507740:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762458.507742:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762458.507745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496137344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762458.507749:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496137472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762458.507780:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762458.507800:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762458/real 1789762458] req@ffff8dd203147800 x1876699496137472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762458.507811:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496137472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762458.507818:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762458.507820:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762458.507823:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496137472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762458.507828:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496137600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762458.507844:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762458.507851:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762458/real 1789762458] req@ffff8dd203146300 x1876699496137600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762458.507860:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496137600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762458.507872:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762458.507875:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762458.507877:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496137600:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762459.083373:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762460.107383:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762461.131306:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762462.155381:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762463.179374:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762463.627243:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762463.627252:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762463.627255:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762463.627264:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2443 00000100:00080000:3.0:1789762463.627274:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762463.627280:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762463.627314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496137728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762463.627341:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762463.627344:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762463.627345:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762463.627349:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2443 00000100:00080000:3.0:1789762463.627351:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762463.627353:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762463.627361:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762463.627370:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496137856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762463.627378:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762463.627385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762463.627387:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762463.627389:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2443 00000100:00080000:3.0:1789762463.627392:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762463.627394:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762463.627397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762463.627403:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496137984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762463.627412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762463.627414:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762463.627416:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762463.627419:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2443 00000100:00080000:3.0:1789762463.627421:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762463.627424:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762463.627427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762463.627436:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496137728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762463.627436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496138112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762463.627519:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762463.627564:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762463/real 1789762463] req@ffff8dd203144a80 x1876699496137728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762463.627585:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496137728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762463.627596:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762463.627601:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762463.627605:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496137728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762463.627613:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496137856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762463.627634:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762463.627645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762463/real 1789762463] req@ffff8dd203144700 x1876699496137856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762463.627657:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496137856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762463.627665:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762463.627668:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762463.627671:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496137856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762463.627677:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496137984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762463.627693:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762463.627702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762463/real 1789762463] req@ffff8dd203145f80 x1876699496137984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762463.627714:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496137984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762463.627722:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762463.627756:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762463.627762:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496137984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762463.627814:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496138112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762463.627843:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762463.627852:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762463/real 1789762463] req@ffff8dd203144380 x1876699496138112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762463.627860:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496138112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762463.627866:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762463.627868:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762463.627870:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496138112:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762464.203388:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762465.227391:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762466.251439:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762467.275318:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762468.299329:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762468.747276:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762468.747287:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762468.747291:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762468.747314:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2448 00000100:00080000:3.0:1789762468.747327:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762468.747334:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762468.747391:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496138240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762468.747417:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762468.747420:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762468.747421:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762468.747436:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2448 00000100:00080000:3.0:1789762468.747442:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762468.747445:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762468.747455:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762468.747466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496138368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762468.747474:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762468.747478:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762468.747479:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762468.747483:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2448 00000100:00080000:3.0:1789762468.747485:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762468.747502:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496138240:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762468.747502:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762468.747507:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762468.747515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496138496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762468.747522:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762468.747525:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762468.747526:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762468.747529:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2448 00000100:00080000:3.0:1789762468.747530:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762468.747533:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762468.747536:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762468.747546:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496138624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762468.747590:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762468.747651:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762468/real 1789762468] req@ffff8dd203145500 x1876699496138240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762468.747667:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496138240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762468.747677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762468.747680:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762468.747684:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496138240:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762468.747705:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496138368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762468.747726:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762468.747737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762468/real 1789762468] req@ffff8dd203146a00 x1876699496138368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762468.747755:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496138368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762468.747762:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762468.747764:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762468.747766:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496138368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762468.747789:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496138496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762468.747803:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762468.747810:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762468/real 1789762468] req@ffff8dd203144e00 x1876699496138496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762468.747818:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496138496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762468.747828:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762468.747830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762468.747832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496138496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762468.747835:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496138624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762468.747846:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762468.747852:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762468/real 1789762468] req@ffff8dd203146d80 x1876699496138624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762468.747858:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496138624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762468.747864:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762468.747866:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762468.747867:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496138624:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762469.323340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762470.347358:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762471.371378:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762472.395368:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762473.419345:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762473.867377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762473.867391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762473.867413:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762473.867423:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2453 00000100:00080000:2.0:1789762473.867436:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762473.867444:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762473.867506:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496138752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762473.867554:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762473.867559:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762473.867561:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762473.867566:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2453 00000100:00080000:2.0:1789762473.867569:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762473.867573:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762473.867585:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762473.867616:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496138880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762473.867635:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762473.867637:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496138752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762473.867641:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762473.867646:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762473.867670:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2453 00000100:00080000:2.0:1789762473.867673:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762473.867677:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762473.867683:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762473.867712:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496139008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762473.867719:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762473.867722:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762473.867724:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762473.867726:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2453 00000100:00080000:2.0:1789762473.867729:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762473.867732:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762473.867735:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762473.867744:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496139136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762473.867763:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762473.867819:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762473/real 1789762473] req@ffff8dd205264700 x1876699496138752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762473.867836:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496138752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762473.867846:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762473.867849:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762473.867852:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496138752:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762473.867878:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496138880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762473.867894:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762473.867902:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762473/real 1789762473] req@ffff8dd205266300 x1876699496138880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762473.867921:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496138880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762473.867927:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762473.867930:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762473.867932:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496138880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762473.867936:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496139008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762473.867946:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762473.867970:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762473/real 1789762473] req@ffff8dd205265180 x1876699496139008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762473.867978:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496139008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762473.867984:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762473.867986:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762473.867988:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496139008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762473.867993:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496139136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762473.868004:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762473.868010:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762473/real 1789762473] req@ffff8dd205266d80 x1876699496139136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762473.868030:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496139136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762473.868036:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762473.868037:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762473.868039:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496139136:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762474.443331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762475.467381:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762476.491341:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762477.515308:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762478.539356:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762478.987284:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762478.987296:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762478.987300:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762478.987311:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2458 00000100:00080000:0.0:1789762478.987322:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762478.987329:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762478.987364:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496139264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762478.987395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762478.987398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762478.987400:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762478.987404:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2458 00000100:00080000:0.0:1789762478.987408:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762478.987411:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762478.987419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762478.987426:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496139264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762478.987432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496139392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762478.987447:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762478.987449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762478.987451:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762478.987454:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2458 00000100:00080000:0.0:1789762478.987456:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762478.987469:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762478.987474:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762478.987481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496139520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762478.987489:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762478.987492:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762478.987493:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762478.987496:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2458 00000100:00080000:0.0:1789762478.987498:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762478.987501:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762478.987503:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762478.987511:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496139648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762478.987542:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496139392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762478.987568:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496139520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762478.987583:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496139648:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762479.563305:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762480.587302:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762481.611285:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762482.059427:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789762482.059464:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762482.059477:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762482.059506:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762482.059512:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762482.059518:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1789762482.059601:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762478/real 1789762482] req@ffff8dd211837100 x1876699496139392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762482.059629:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496139392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762482.059646:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762482.059651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762482.059656:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496139392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762482.059664:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762478/real 1789762482] req@ffff8dd211836300 x1876699496139520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762482.059677:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496139520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762482.059685:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762482.059689:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762482.059693:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496139520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762482.059700:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762478/real 1789762482] req@ffff8dd211837800 x1876699496139648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762482.059718:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496139648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762482.059727:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762482.059730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762482.059733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496139648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762482.059740:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762478/real 1789762482] req@ffff8dd211837b80 x1876699496139264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762482.059752:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496139264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762482.059760:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762482.059764:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762482.059768:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496139264:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789762482.635285:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762483.659301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762484.107251:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762484.107259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762484.107262:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762484.107272:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2463 00000100:00080000:2.0:1789762484.107280:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762484.107285:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762484.107314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496139776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762484.107337:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762484.107340:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762484.107342:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762484.107346:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2463 00000100:00080000:2.0:1789762484.107350:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762484.107355:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762484.107366:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762484.107369:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496139776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762484.107384:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496139904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762484.107397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762484.107401:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762484.107403:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762484.107408:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2463 00000100:00080000:2.0:1789762484.107413:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762484.107417:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762484.107422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762484.107435:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496140032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762484.107439:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762484.107450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762484.107453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762484.107454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762484.107457:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2463 00000100:00080000:2.0:1789762484.107459:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762484.107462:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762484.107466:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762484.107473:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496140160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762484.107493:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762484/real 1789762484] req@ffff8dd205265f80 x1876699496139776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762484.107521:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496139776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762484.107532:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762484.107537:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762484.107541:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496139776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762484.107565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496139904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762484.107603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762484.107622:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762484/real 1789762484] req@ffff8dd205266d80 x1876699496139904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762484.107637:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496139904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762484.107646:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762484.107648:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762484.107651:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496139904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762484.107655:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496140032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762484.107672:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762484.107683:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762484/real 1789762484] req@ffff8dd205265180 x1876699496140032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762484.107708:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496140032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762484.107724:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762484.107728:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762484.107732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496140032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762484.107738:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496140160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762484.107761:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762484.107773:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762484/real 1789762484] req@ffff8dd205266300 x1876699496140160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762484.107786:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496140160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762484.107796:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762484.107800:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762484.107804:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496140160:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762484.683304:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762485.707353:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762486.731297:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762487.755324:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762488.779324:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762489.227264:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762489.227276:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762489.227281:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762489.227292:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2468 00000100:00080000:0.0:1789762489.227304:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762489.227313:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762489.227354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496140288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762489.227385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762489.227389:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762489.227392:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762489.227398:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2468 00000100:00080000:0.0:1789762489.227403:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762489.227407:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762489.227416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762489.227427:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496140288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762489.227436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496140416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762489.227452:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762489.227456:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762489.227459:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762489.227464:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2468 00000100:00080000:0.0:1789762489.227468:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762489.227473:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762489.227480:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762489.227494:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496140544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762489.227506:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762489.227510:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762489.227512:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762489.227521:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2468 00000800:00000100:2.0:1789762489.227523:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789762489.227529:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762489.227548:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762489.227554:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762489.227566:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496140672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762489.227598:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762489/real 1789762489] req@ffff8dd211837b80 x1876699496140288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762489.227618:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496140288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762489.227627:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762489.227631:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762489.227634:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496140288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762489.227659:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496140416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762489.227680:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762489.227690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762489/real 1789762489] req@ffff8dd211837800 x1876699496140416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762489.227699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496140416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762489.227705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762489.227708:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762489.227711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496140416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762489.227715:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496140544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762489.227728:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762489.227734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762489/real 1789762489] req@ffff8dd211836300 x1876699496140544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762489.227741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496140544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762489.227748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762489.227750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762489.227752:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496140544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762489.227756:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496140672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762489.227766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762489.227772:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762489/real 1789762489] req@ffff8dd211837100 x1876699496140672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762489.227779:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496140672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762489.227785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762489.227787:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762489.227789:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496140672:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762489.803307:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762490.827304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762491.851335:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762492.875351:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762493.899282:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762494.347277:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762494.347286:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762494.347289:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762494.347297:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2473 00000100:00080000:3.0:1789762494.347307:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762494.347315:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762494.347353:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496140800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762494.347377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762494.347379:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762494.347381:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762494.347384:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2473 00000100:00080000:3.0:1789762494.347387:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762494.347390:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762494.347398:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762494.347411:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496140928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762494.347420:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496140800:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762494.347422:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762494.347428:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762494.347432:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762494.347436:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2473 00000100:00080000:3.0:1789762494.347438:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762494.347440:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762494.347443:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762494.347452:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496141056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762494.347460:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762494.347462:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762494.347464:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762494.347466:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2473 00000100:00080000:3.0:1789762494.347480:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762494.347483:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762494.347486:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762494.347506:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496141184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762494.347513:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762494.347565:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762494/real 1789762494] req@ffff8dd203147b80 x1876699496140800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762494.347580:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496140800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762494.347589:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762494.347592:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762494.347595:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496140800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762494.347614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496140928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762494.347633:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762494.347642:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762494/real 1789762494] req@ffff8dd203146d80 x1876699496140928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762494.347650:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496140928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762494.347656:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762494.347658:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762494.347660:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496140928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762494.347664:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496141056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762494.347678:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762494.347684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762494/real 1789762494] req@ffff8dd203144e00 x1876699496141056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762494.347691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496141056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762494.347696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762494.347698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762494.347700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496141056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762494.347703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496141184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762494.347713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762494.347719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762494/real 1789762494] req@ffff8dd203146a00 x1876699496141184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762494.347726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496141184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762494.347731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762494.347733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762494.347735:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496141184:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762494.923283:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762495.947307:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762496.971293:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762497.995334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762499.019306:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762499.467245:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762499.467257:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762499.467261:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762499.467272:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2479 00000100:00080000:3.0:1789762499.467284:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762499.467292:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762499.467329:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496141312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762499.467356:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762499.467360:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762499.467362:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762499.467371:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2479 00000100:00080000:3.0:1789762499.467376:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762499.467380:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762499.467390:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762499.467405:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496141440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762499.467408:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496141312:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762499.467429:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762499.467433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762499.467436:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762499.467441:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2479 00000100:00080000:3.0:1789762499.467446:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762499.467450:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762499.467470:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762499.467486:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496141568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762499.467488:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762499.467502:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762499.467506:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762499.467508:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762499.467513:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2479 00000100:00080000:3.0:1789762499.467518:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762499.467523:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762499.467529:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762499.467537:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762499/real 1789762499] req@ffff8dd203145500 x1876699496141312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762499.467543:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496141696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762499.467559:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496141312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762499.467569:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762499.467573:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762499.467577:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496141312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762499.467596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496141440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762499.467612:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762499.467620:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762499/real 1789762499] req@ffff8dd203144380 x1876699496141440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762499.467628:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496141440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762499.467633:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762499.467635:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762499.467638:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496141440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762499.467641:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496141568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762499.467652:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762499.467657:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762499/real 1789762499] req@ffff8dd203145f80 x1876699496141568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762499.467669:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496141568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762499.467674:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762499.467676:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762499.467678:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496141568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762499.467681:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496141696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762499.467697:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762499.467707:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762499/real 1789762499] req@ffff8dd203144700 x1876699496141696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762499.467720:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496141696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762499.467728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762499.467731:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762499.467735:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496141696:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762500.043293:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762501.067299:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762502.091282:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762503.115303:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762504.139303:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762504.587267:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762504.587275:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762504.587279:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762504.587287:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2484 00000100:00080000:0.0:1789762504.587297:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762504.587302:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762504.587334:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496141824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762504.587356:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762504.587358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762504.587360:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762504.587363:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2484 00000100:00080000:0.0:1789762504.587365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762504.587368:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762504.587376:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762504.587384:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496141952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762504.587393:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496141824:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762504.587394:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762504.587402:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762504.587404:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762504.587407:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2484 00000100:00080000:0.0:1789762504.587409:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762504.587412:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762504.587415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762504.587443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496142080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762504.587450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762504.587453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762504.587454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762504.587457:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2484 00000100:00080000:0.0:1789762504.587460:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762504.587463:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762504.587466:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762504.587475:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496142208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762504.587485:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762504.587533:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762504/real 1789762504] req@ffff8dd211834700 x1876699496141824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762504.587555:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496141824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762504.587563:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762504.587568:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762504.587571:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496141824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762504.587596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496141952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762504.587622:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762504.587636:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762504/real 1789762504] req@ffff8dd211835c00 x1876699496141952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762504.587649:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496141952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762504.587659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762504.587663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762504.587667:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496141952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762504.587672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496142080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762504.587689:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762504.587697:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762504/real 1789762504] req@ffff8dd211836a00 x1876699496142080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762504.587706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496142080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762504.587724:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762504.587726:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762504.587728:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496142080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762504.587732:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496142208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762504.587744:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762504.587751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762504/real 1789762504] req@ffff8dd211834000 x1876699496142208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762504.587758:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496142208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762504.587769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762504.587771:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762504.587773:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496142208:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762505.163300:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762506.187311:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762507.211311:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762508.235325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762509.259339:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762509.707262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762509.707270:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762509.707274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762509.707283:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2489 00000100:00080000:1.0:1789762509.707293:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762509.707300:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762509.707333:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496142336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762509.707354:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762509.707357:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762509.707358:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762509.707361:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2489 00000100:00080000:1.0:1789762509.707363:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762509.707365:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762509.707373:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762509.707395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496142464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762509.707402:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762509.707404:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762509.707405:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762509.707408:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2489 00000100:00080000:1.0:1789762509.707410:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762509.707412:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762509.707415:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762509.707422:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496142592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762509.707429:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762509.707431:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762509.707433:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762509.707436:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2489 00000100:00080000:1.0:1789762509.707438:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762509.707444:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762509.707446:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496142336:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789762509.707447:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762509.707458:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496142720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762509.707505:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762509.707538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762509/real 1789762509] req@ffff8dd20504c000 x1876699496142336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762509.707551:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496142336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762509.707563:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762509.707566:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762509.707569:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496142336:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762509.707574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496142464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762509.707588:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762509.707595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762509/real 1789762509] req@ffff8dd20504d880 x1876699496142464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762509.707603:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496142464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762509.707608:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762509.707610:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762509.707612:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496142464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762509.707616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496142592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762509.707626:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762509.707631:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762509/real 1789762509] req@ffff8dd20504ed80 x1876699496142592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762509.707638:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496142592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762509.707644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762509.707645:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762509.707647:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496142592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762509.707668:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496142720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762509.707679:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762509.707685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762509/real 1789762509] req@ffff8dd20504f100 x1876699496142720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762509.707692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496142720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762509.707697:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762509.707699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762509.707700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496142720:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762510.283285:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762511.307298:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762512.331279:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762513.355361:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762514.379354:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762514.827287:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762514.827300:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762514.827304:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762514.827316:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2494 00000100:00080000:1.0:1789762514.827329:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762514.827337:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762514.827406:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496142848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762514.827437:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762514.827440:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762514.827442:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762514.827446:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2494 00000100:00080000:1.0:1789762514.827450:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762514.827454:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762514.827465:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762514.827482:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496142976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762514.827499:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496142848:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762514.827525:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762514.827532:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762514.827537:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762514.827545:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2494 00000100:00080000:1.0:1789762514.827552:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762514.827558:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762514.827567:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789762514.827579:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762514.827586:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496143104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762514.827599:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762514.827602:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762514.827604:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762514.827611:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2494 00000100:00080000:1.0:1789762514.827617:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762514.827622:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762514.827626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762514/real 1789762514] req@ffff8dd20504ce00 x1876699496142848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762514.827629:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762514.827653:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496142848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762514.827658:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496143232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762514.827687:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762514.827696:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762514.827703:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496142848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762514.827736:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496142976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762514.827762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762514.827772:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762514/real 1789762514] req@ffff8dd20504c700 x1876699496142976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762514.827781:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496142976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762514.827787:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762514.827790:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762514.827792:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496142976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762514.827797:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496143104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762514.827810:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762514.827817:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762514/real 1789762514] req@ffff8dd20504f800 x1876699496143104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762514.827824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496143104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762514.827830:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762514.827831:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762514.827833:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496143104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762514.827837:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496143232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762514.827848:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762514.827854:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762514/real 1789762514] req@ffff8dd20504fb80 x1876699496143232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762514.827862:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496143232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762514.827867:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762514.827869:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762514.827870:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496143232:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762515.403380:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762516.427383:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762517.451317:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762518.475289:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762519.499287:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762519.947256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762519.947264:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762519.947268:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762519.947277:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2499 00000100:00080000:2.0:1789762519.947286:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762519.947292:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762519.947340:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496143360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762519.947363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762519.947367:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762519.947369:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762519.947374:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2499 00000100:00080000:2.0:1789762519.947379:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762519.947384:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762519.947393:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496143360:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789762519.947395:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762519.947424:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496143488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762519.947433:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762519.947436:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762519.947438:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762519.947441:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2499 00000100:00080000:2.0:1789762519.947445:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762519.947447:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789762519.947452:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789762519.947456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762519.947468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496143616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762519.947476:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762519.947479:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762519.947480:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762519.947482:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2499 00000100:00080000:2.0:1789762519.947485:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762519.947488:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762519.947492:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762519/real 1789762519] req@ffff8dd205264700 x1876699496143360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762519.947493:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762519.947508:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496143744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762519.947514:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496143360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762519.947526:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762519.947530:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762519.947534:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496143360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762519.947553:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496143488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762519.947568:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762519.947576:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762519/real 1789762519] req@ffff8dd205264000 x1876699496143488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762519.947584:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496143488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762519.947590:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762519.947592:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762519.947594:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496143488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762519.947598:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496143616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762519.947614:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762519.947625:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762519/real 1789762519] req@ffff8dd205267b80 x1876699496143616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762519.947636:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496143616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762519.947644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762519.947647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762519.947651:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496143616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762519.947657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496143744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762519.947677:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762519.947689:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762519/real 1789762519] req@ffff8dd205266a00 x1876699496143744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762519.947701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496143744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762519.947709:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762519.947713:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762519.947716:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496143744:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762520.523287:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762521.547308:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762522.571329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762523.595300:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762524.619358:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762525.067277:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762525.067286:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762525.067289:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762525.067298:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2504 00000100:00080000:2.0:1789762525.067307:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762525.067313:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762525.067344:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496143872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762525.067368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762525.067370:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762525.067372:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762525.067374:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2504 00000100:00080000:2.0:1789762525.067377:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762525.067379:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762525.067386:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762525.067395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496144000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762525.067402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762525.067405:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762525.067406:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762525.067409:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2504 00000100:00080000:2.0:1789762525.067411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762525.067413:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762525.067417:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762525.067420:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496143872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762525.067427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496144128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762525.067434:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762525.067436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762525.067437:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762525.067440:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2504 00000100:00080000:2.0:1789762525.067442:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762525.067444:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762525.067447:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762525.067455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496144256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762525.067483:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762525.067532:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762525/real 1789762525] req@ffff8dd205265880 x1876699496143872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762525.067548:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496143872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762525.067556:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762525.067566:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762525.067570:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496143872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762525.067579:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496144000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762525.067604:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762525.067618:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762525/real 1789762525] req@ffff8dd205267100 x1876699496144000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762525.067630:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496144000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762525.067640:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762525.067642:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762525.067645:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496144000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762525.067663:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496144128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762525.067678:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762525.067685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762525/real 1789762525] req@ffff8dd205264e00 x1876699496144128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762525.067692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496144128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762525.067697:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762525.067700:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762525.067702:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496144128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762525.067705:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496144256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762525.067714:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762525.067720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762525/real 1789762525] req@ffff8dd205264380 x1876699496144256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762525.067727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496144256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762525.067732:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762525.067734:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762525.067736:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496144256:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762525.643290:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762526.667343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762527.691323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762528.715407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762529.739296:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762530.187261:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762530.187272:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762530.187276:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762530.187287:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2509 00000100:00080000:2.0:1789762530.187298:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762530.187306:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762530.187343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496144384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762530.187364:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762530.187367:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762530.187368:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762530.187371:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2509 00000100:00080000:2.0:1789762530.187374:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762530.187378:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762530.187387:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762530.187400:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496144512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762530.187401:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496144384:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762530.187416:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762530.187418:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762530.187419:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762530.187421:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2509 00000100:00080000:2.0:1789762530.187423:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762530.187425:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762530.187429:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762530.187442:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496144640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762530.187452:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762530.187455:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762530.187457:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762530.187461:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2509 00000100:00080000:2.0:1789762530.187465:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762530.187471:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762530.187475:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762530.187483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762530.187496:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496144768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762530.187512:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762530/real 1789762530] req@ffff8dd205267800 x1876699496144384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762530.187532:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496144384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762530.187543:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762530.187547:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762530.187551:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496144384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762530.187574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496144512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762530.187596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762530.187609:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762530/real 1789762530] req@ffff8dd205264a80 x1876699496144512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762530.187621:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496144512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762530.187631:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762530.187634:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762530.187638:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496144512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762530.187644:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496144640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762530.187662:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762530.187673:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762530/real 1789762530] req@ffff8dd205267480 x1876699496144640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762530.187686:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496144640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762530.187694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762530.187697:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762530.187700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496144640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762530.187706:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496144768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762530.187722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762530.187732:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762530/real 1789762530] req@ffff8dd205266680 x1876699496144768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762530.187743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496144768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762530.187751:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762530.187754:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762530.187757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496144768:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762530.763405:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762531.787295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762532.811345:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762533.835320:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762534.859319:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762535.307268:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762535.307277:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762535.307282:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762535.307293:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2514 00000100:00080000:2.0:1789762535.307312:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762535.307321:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762535.307366:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496144896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762535.307401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762535.307405:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762535.307408:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762535.307412:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2514 00000100:00080000:2.0:1789762535.307417:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762535.307421:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762535.307430:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762535.307447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496145024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762535.307459:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762535.307465:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762535.307467:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762535.307472:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2514 00000100:00080000:2.0:1789762535.307485:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762535.307493:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762535.307502:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496144896:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789762535.307505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762535.307530:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496145152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762535.307543:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762535.307545:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762535.307547:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762535.307556:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2514 00000100:00080000:2.0:1789762535.307560:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762535.307563:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762535.307567:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762535.307577:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496145280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762535.307608:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762535.307675:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762535/real 1789762535] req@ffff8dd205265500 x1876699496144896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762535.307699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496144896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762535.307714:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762535.307719:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762535.307724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496144896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762535.307732:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496145024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762535.307756:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762535.307769:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762535/real 1789762535] req@ffff8dd205266680 x1876699496145024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762535.307782:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496145024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762535.307790:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762535.307794:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762535.307797:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496145024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762535.307828:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496145152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762535.307848:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762535.307859:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762535/real 1789762535] req@ffff8dd205267480 x1876699496145152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762535.307872:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496145152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762535.307880:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762535.307883:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762535.307887:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496145152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762535.307894:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496145280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762535.307911:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762535.307921:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762535/real 1789762535] req@ffff8dd205264a80 x1876699496145280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762535.307933:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496145280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762535.307941:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762535.307945:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762535.307949:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496145280:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762535.883343:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762536.907337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762537.931341:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762538.955351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762539.979295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762540.427262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762540.427275:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762540.427281:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762540.427295:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2520 00000100:00080000:2.0:1789762540.427310:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762540.427320:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762540.427367:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496145408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762540.427399:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762540.427403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762540.427405:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762540.427410:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2520 00000100:00080000:2.0:1789762540.427414:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762540.427418:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762540.427431:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762540.427449:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496145536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762540.427452:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496145408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762540.427472:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762540.427477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762540.427479:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762540.427486:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2520 00000100:00080000:2.0:1789762540.427490:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762540.427495:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762540.427501:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762540.427516:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496145664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762540.427536:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762540.427539:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762540.427545:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762540.427549:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762540.427557:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2520 00000100:00080000:2.0:1789762540.427563:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762540.427568:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762540.427574:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762540.427590:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496145792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762540.427602:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762540/real 1789762540] req@ffff8dd205267800 x1876699496145408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762540.427627:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496145408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762540.427639:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762540.427644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762540.427649:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496145408:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762540.427672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496145536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762540.427696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762540.427708:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762540/real 1789762540] req@ffff8dd205264380 x1876699496145536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762540.427723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496145536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762540.427731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762540.427735:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762540.427739:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496145536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762540.427745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496145664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762540.427762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762540.427771:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762540/real 1789762540] req@ffff8dd205264e00 x1876699496145664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762540.427784:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496145664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762540.427792:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762540.427796:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762540.427800:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496145664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762540.427806:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496145792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762540.427822:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762540.427831:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762540/real 1789762540] req@ffff8dd205267100 x1876699496145792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762540.427843:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496145792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762540.427858:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762540.427861:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762540.427865:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496145792:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762541.003397:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762542.027326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762543.051289:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762544.075331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762545.099310:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762545.547245:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762545.547253:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762545.547256:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762545.547265:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2525 00000100:00080000:2.0:1789762545.547274:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762545.547281:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762545.547312:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496145920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762545.547334:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762545.547336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762545.547338:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762545.547341:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2525 00000100:00080000:2.0:1789762545.547343:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762545.547345:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762545.547353:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762545.547360:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496146048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762545.547368:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762545.547370:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762545.547372:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762545.547375:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2525 00000100:00080000:2.0:1789762545.547377:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762545.547379:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762545.547382:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762545.547388:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496146176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762545.547393:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496145920:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762545.547397:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762545.547401:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762545.547402:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762545.547405:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2525 00000100:00080000:2.0:1789762545.547407:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762545.547409:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762545.547412:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762545.547420:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496146304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762545.547486:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496146048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762545.547522:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496146176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762545.547544:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496146304:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762546.123343:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762547.147326:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762548.171310:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762548.619519:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789762548.619572:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762548.619593:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762548.619645:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762548.619655:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762548.619664:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789762548.619752:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762545/real 1789762548] req@ffff8dd205267b80 x1876699496146176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762548.619801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496146176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762548.619824:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762548.619834:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762548.619842:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496146176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762548.619870:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762545/real 1789762548] req@ffff8dd205264000 x1876699496146304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762548.619905:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496146304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762548.619923:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762548.619931:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762548.619937:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496146304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762548.619951:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762545/real 1789762548] req@ffff8dd205265880 x1876699496145920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762548.619976:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496145920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762548.619990:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762548.619996:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762548.620002:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496145920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762548.620014:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762545/real 1789762548] req@ffff8dd205266a00 x1876699496146048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762548.620038:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496146048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762548.620054:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762548.620061:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762548.620067:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496146048:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762549.195286:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762550.219351:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762550.667241:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762550.667249:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762550.667253:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762550.667262:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2530 00000100:00080000:2.0:1789762550.667271:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762550.667277:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762550.667306:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496146432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762550.667333:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762550.667337:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762550.667339:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762550.667343:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2530 00000100:00080000:2.0:1789762550.667347:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762550.667352:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762550.667362:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762550.667376:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496146432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762550.667379:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496146560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762550.667393:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762550.667396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762550.667399:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762550.667404:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2530 00000100:00080000:2.0:1789762550.667407:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762550.667411:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762550.667417:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762550.667430:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496146688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762550.667440:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762550.667445:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789762550.667446:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762550.667448:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762550.667454:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2530 00000100:00080000:2.0:1789762550.667459:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762550.667463:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762550.667478:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762550.667490:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762550/real 1789762550] req@ffff8dd205264700 x1876699496146432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762550.667497:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496146816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762550.667528:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496146432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762550.667542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762550.667546:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762550.667550:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496146432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762550.667574:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496146560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762550.667600:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762550.667614:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762550/real 1789762550] req@ffff8dd205266300 x1876699496146560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762550.667628:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496146560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762550.667638:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762550.667641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762550.667645:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496146560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762550.667652:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496146688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762550.667674:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762550.667685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762550/real 1789762550] req@ffff8dd205265180 x1876699496146688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762550.667697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496146688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762550.667706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762550.667710:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762550.667713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496146688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762550.667720:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496146816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762550.667746:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762550.667759:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762550/real 1789762550] req@ffff8dd205266d80 x1876699496146816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762550.667772:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496146816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762550.667781:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762550.667784:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762550.667788:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496146816:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762551.243368:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762552.267320:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762553.291317:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762554.315410:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762555.339297:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762555.787282:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762555.787294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762555.787298:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762555.787309:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2535 00000100:00080000:3.0:1789762555.787320:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762555.787328:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762555.787370:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496146944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762555.787391:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762555.787394:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762555.787395:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762555.787398:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2535 00000100:00080000:3.0:1789762555.787401:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762555.787403:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762555.787416:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762555.787430:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496147072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762555.787439:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496146944:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762555.787445:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762555.787451:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762555.787453:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762555.787458:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2535 00000100:00080000:3.0:1789762555.787463:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762555.787468:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762555.787474:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762555.787484:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496147200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762555.787492:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762555.787495:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762555.787496:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762555.787498:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2535 00000100:00080000:3.0:1789762555.787502:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762555.787507:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762555.787510:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762555.787514:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762555.787524:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496147328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762555.787546:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762555/real 1789762555] req@ffff8dd203144a80 x1876699496146944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762555.787560:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496146944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762555.787568:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762555.787570:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762555.787573:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496146944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762555.787590:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496147072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762555.787605:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762555.787613:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762555/real 1789762555] req@ffff8dd203146300 x1876699496147072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762555.787622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496147072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762555.787628:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762555.787630:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762555.787632:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496147072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762555.787636:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496147200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762555.787646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762555.787652:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762555/real 1789762555] req@ffff8dd203147800 x1876699496147200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762555.787659:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496147200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762555.787664:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762555.787666:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762555.787668:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496147200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762555.787671:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496147328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762555.787681:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762555.787686:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762555/real 1789762555] req@ffff8dd203145180 x1876699496147328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762555.787693:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496147328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762555.787702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762555.787704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762555.787706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496147328:192.168.203.109@tcp:38:kworker.0 00010000:00010000:3.0:1789762556.363288:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762557.387422:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762558.411311:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762559.435330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762560.459260:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762560.907255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762560.907267:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762560.907271:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762560.907282:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2540 00000100:00080000:3.0:1789762560.907293:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762560.907300:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762560.907337:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496147456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762560.907363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762560.907366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762560.907368:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762560.907371:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2540 00000100:00080000:3.0:1789762560.907374:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762560.907376:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762560.907383:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762560.907393:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496147584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762560.907406:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762560.907410:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789762560.907412:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496147456:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762560.907413:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762560.907421:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2540 00000100:00080000:3.0:1789762560.907423:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762560.907426:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762560.907429:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762560.907438:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496147712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762560.907445:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762560.907448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762560.907449:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762560.907452:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2540 00000100:00080000:3.0:1789762560.907462:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762560.907464:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762560.907467:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762560.907473:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762560.907476:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496147840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762560.907519:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762560/real 1789762560] req@ffff8dd203145c00 x1876699496147456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762560.907533:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496147456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762560.907540:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762560.907543:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762560.907546:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496147456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762560.907564:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496147584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762560.907580:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762560.907588:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762560/real 1789762560] req@ffff8dd203147480 x1876699496147584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762560.907596:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496147584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762560.907602:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762560.907604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762560.907606:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496147584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762560.907610:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496147712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762560.907621:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762560.907627:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762560/real 1789762560] req@ffff8dd203144000 x1876699496147712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762560.907634:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496147712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762560.907644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762560.907646:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762560.907648:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496147712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762560.907651:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496147840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762560.907660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762560.907666:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762560/real 1789762560] req@ffff8dd203147100 x1876699496147840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762560.907673:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496147840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762560.907681:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762560.907684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762560.907687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496147840:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762561.483318:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762562.507299:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762563.531313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762564.555256:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762565.579325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762566.027276:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762566.027285:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762566.027288:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762566.027297:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2545 00000100:00080000:1.0:1789762566.027306:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762566.027311:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762566.027347:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496147968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762566.027375:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762566.027379:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762566.027381:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762566.027386:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2545 00000100:00080000:1.0:1789762566.027390:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762566.027395:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762566.027405:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762566.027413:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496147968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762566.027423:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496148096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762566.027435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762566.027438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762566.027440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762566.027443:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2545 00000100:00080000:1.0:1789762566.027446:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762566.027448:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762566.027453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762566.027463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496148224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762566.027470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762566.027472:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762566.027474:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762566.027479:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762566.027480:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2545 00000100:00080000:1.0:1789762566.027485:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762566.027488:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762566.027491:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762566.027499:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496148352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762566.027529:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762566/real 1789762566] req@ffff8dd20504df80 x1876699496147968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762566.027543:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496147968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762566.027551:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762566.027555:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762566.027557:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496147968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762566.027576:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496148096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762566.027594:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762566.027604:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762566/real 1789762566] req@ffff8dd20504d180 x1876699496148096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762566.027617:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496148096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762566.027625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762566.027629:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762566.027632:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496148096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762566.027639:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496148224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762566.027660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762566.027668:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762566/real 1789762566] req@ffff8dd20504e300 x1876699496148224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762566.027676:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496148224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762566.027682:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762566.027684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762566.027686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496148224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762566.027689:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496148352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762566.027701:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762566.027707:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762566/real 1789762566] req@ffff8dd20504d500 x1876699496148352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762566.027715:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496148352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762566.027719:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762566.027721:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762566.027723:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496148352:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762566.603283:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762567.627322:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762568.651303:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762569.675307:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762570.699331:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762571.147276:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762571.147285:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762571.147288:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762571.147297:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2550 00000100:00080000:1.0:1789762571.147306:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762571.147311:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762571.147344:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496148480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762571.147371:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762571.147375:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762571.147377:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762571.147383:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2550 00000100:00080000:1.0:1789762571.147388:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762571.147393:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762571.147403:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762571.147403:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496148480:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762571.147427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496148608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762571.147437:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762571.147440:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762571.147441:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762571.147445:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2550 00000100:00080000:1.0:1789762571.147448:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762571.147451:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762571.147456:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789762571.147466:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762571.147467:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496148736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762571.147481:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762571.147484:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762571.147486:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762571.147488:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2550 00000100:00080000:1.0:1789762571.147498:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762571.147503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762571.147504:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762571/real 1789762571] req@ffff8dd20504e680 x1876699496148480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762571.147507:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762571.147519:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496148864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762571.147529:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496148480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762571.147542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762571.147548:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762571.147552:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496148480:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762571.147576:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496148608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762571.147602:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762571.147631:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762571/real 1789762571] req@ffff8dd20504d500 x1876699496148608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762571.147645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496148608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762571.147654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762571.147658:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762571.147662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496148608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762571.147668:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496148736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762571.147688:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762571.147697:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762571/real 1789762571] req@ffff8dd20504e300 x1876699496148736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762571.147705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496148736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762571.147710:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762571.147713:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762571.147715:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496148736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762571.147718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496148864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762571.147729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762571.147734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762571/real 1789762571] req@ffff8dd20504d180 x1876699496148864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762571.147741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496148864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762571.147747:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762571.147749:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762571.147752:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496148864:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762571.723292:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762572.747276:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762573.771329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762574.795305:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762575.819387:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762576.267367:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762576.267375:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762576.267379:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762576.267387:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2555 00000100:00080000:2.0:1789762576.267396:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762576.267401:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762576.267439:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496148992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762576.267467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762576.267470:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762576.267471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762576.267474:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2555 00000100:00080000:2.0:1789762576.267476:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762576.267478:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762576.267485:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762576.267494:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496149120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762576.267501:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762576.267503:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762576.267505:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762576.267508:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2555 00000100:00080000:2.0:1789762576.267510:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762576.267512:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762576.267515:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762576.267520:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496149248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762576.267526:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762576.267527:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762576.267528:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762576.267531:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2555 00000100:00080000:2.0:1789762576.267534:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762576.267537:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762576.267539:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762576.267545:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496149376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762576.267594:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496148992:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1789762576.267702:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762576.267801:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762576/real 1789762576] req@ffff8dd205265f80 x1876699496148992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762576.267832:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496148992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762576.267848:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762576.267854:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762576.267859:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496148992:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762576.267870:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496149120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762576.267931:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762576.267976:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762576/real 1789762576] req@ffff8dd205266d80 x1876699496149120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762576.267999:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496149120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762576.268009:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762576.268012:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762576.268016:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496149120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762576.268023:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496149248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762576.268046:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762576.268056:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762576/real 1789762576] req@ffff8dd205265180 x1876699496149248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762576.268069:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496149248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762576.268078:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762576.268081:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762576.268085:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496149248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762576.268095:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496149376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762576.268124:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762576.268143:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762576/real 1789762576] req@ffff8dd205266300 x1876699496149376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762576.268165:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496149376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762576.268230:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762576.268234:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762576.268238:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496149376:192.168.203.109@tcp:38:kworker.0 00010000:00010000:3.0:1789762576.843323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762577.867355:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762578.891356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762579.915348:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762580.939281:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762581.387247:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762581.387256:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762581.387259:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762581.387270:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2560 00000100:00080000:2.0:1789762581.387280:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762581.387286:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762581.387318:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496149504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762581.387351:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762581.387355:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762581.387357:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762581.387362:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2560 00000100:00080000:2.0:1789762581.387366:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762581.387371:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762581.387381:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762581.387393:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496149632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762581.387402:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762581.387406:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762581.387408:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762581.387414:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496149504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762581.387424:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2560 00000100:00080000:2.0:1789762581.387432:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762581.387435:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762581.387439:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762581.387447:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496149760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762581.387455:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762581.387458:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762581.387459:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762581.387462:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2560 00000100:00080000:2.0:1789762581.387464:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762581.387468:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762581.387471:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762581.387479:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496149888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762581.387518:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762581.387566:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762581/real 1789762581] req@ffff8dd205264700 x1876699496149504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762581.387582:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496149504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762581.387592:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762581.387596:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762581.387599:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496149504:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762581.387625:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496149632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762581.387644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762581.387654:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762581/real 1789762581] req@ffff8dd205266a00 x1876699496149632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762581.387663:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496149632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762581.387670:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762581.387672:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762581.387674:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496149632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762581.387678:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496149760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762581.387690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762581.387697:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762581/real 1789762581] req@ffff8dd205265880 x1876699496149760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762581.387705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496149760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762581.387711:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762581.387712:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762581.387714:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496149760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762581.387718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496149888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762581.387729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762581.387735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762581/real 1789762581] req@ffff8dd205264000 x1876699496149888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762581.387743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496149888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762581.387753:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762581.387755:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762581.387757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496149888:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762581.963345:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762582.987288:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762584.011297:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762585.035282:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762586.059289:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762586.507238:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762586.507247:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762586.507250:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762586.507259:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2566 00000100:00080000:2.0:1789762586.507268:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762586.507274:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762586.507307:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496150016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762586.507334:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762586.507338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762586.507340:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762586.507344:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2566 00000100:00080000:2.0:1789762586.507348:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762586.507352:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762586.507363:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762586.507379:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496150144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762586.507387:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496150016:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762586.507402:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762586.507404:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762586.507406:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762586.507410:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2566 00000100:00080000:2.0:1789762586.507413:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762586.507416:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762586.507420:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762586.507432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496150272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762586.507443:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762586.507447:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789762586.507452:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762586.507452:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762586.507459:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2566 00000100:00080000:2.0:1789762586.507465:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762586.507471:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762586.507477:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762586.507490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496150400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762586.507495:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762586/real 1789762586] req@ffff8dd205267b80 x1876699496150016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762586.507515:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496150016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762586.507524:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762586.507527:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762586.507530:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496150016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762586.507549:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496150144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762586.507565:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762586.507572:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762586/real 1789762586] req@ffff8dd205267100 x1876699496150144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762586.507580:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496150144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762586.507586:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762586.507588:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762586.507591:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496150144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762586.507595:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496150272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762586.507616:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762586.507623:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762586/real 1789762586] req@ffff8dd205264e00 x1876699496150272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762586.507630:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496150272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762586.507635:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762586.507637:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762586.507640:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496150272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762586.507643:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496150400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762586.507653:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762586.507658:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762586/real 1789762586] req@ffff8dd205264380 x1876699496150400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762586.507666:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496150400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762586.507671:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762586.507672:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762586.507674:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496150400:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762587.083412:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762588.107302:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762589.131329:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762590.155351:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762591.179332:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762591.627282:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762591.627292:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762591.627297:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762591.627306:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2571 00000100:00080000:2.0:1789762591.627315:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762591.627321:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762591.627372:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496150528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762591.627396:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762591.627399:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762591.627400:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762591.627403:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2571 00000100:00080000:2.0:1789762591.627405:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762591.627407:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762591.627414:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762591.627422:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496150656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762591.627433:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762591.627438:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762591.627440:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762591.627444:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2571 00000100:00080000:2.0:1789762591.627448:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762591.627454:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762591.627462:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762591.627476:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496150784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762591.627497:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496150528:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762591.627501:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762591.627513:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762591.627514:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762591.627519:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2571 00000100:00080000:2.0:1789762591.627522:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762591.627526:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762591.627533:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762591.627546:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496150912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762591.627618:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762591.627690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762591/real 1789762591] req@ffff8dd205267800 x1876699496150528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762591.627717:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496150528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762591.627733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762591.627739:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762591.627745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496150528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762591.627756:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496150656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762591.627844:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762591.627860:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762591/real 1789762591] req@ffff8dd205264a80 x1876699496150656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762591.627894:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496150656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762591.627912:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762591.627919:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762591.627924:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496150656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762591.627966:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496150784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762591.628000:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762591.628015:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762591/real 1789762591] req@ffff8dd205267480 x1876699496150784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762591.628030:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496150784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762591.628066:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762591.628071:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762591.628075:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496150784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762591.628083:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496150912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762591.628106:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762591.628119:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762591/real 1789762591] req@ffff8dd205266680 x1876699496150912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762591.628133:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496150912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762591.628153:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762591.628157:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762591.628161:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496150912:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762592.203465:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762593.227367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762594.251359:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762595.275311:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762596.299280:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762596.747239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762596.747249:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762596.747253:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762596.747261:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2576 00000100:00080000:2.0:1789762596.747270:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762596.747276:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762596.747308:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496151040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762596.747332:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762596.747336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762596.747338:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762596.747342:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2576 00000100:00080000:2.0:1789762596.747346:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762596.747350:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762596.747360:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762596.747375:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496151168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762596.747385:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762596.747388:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496151040:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762596.747388:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762596.747390:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762596.747393:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2576 00000100:00080000:2.0:1789762596.747396:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762596.747400:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762596.747404:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762596.747414:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496151296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762596.747423:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762596.747425:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762596.747427:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762596.747430:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2576 00000100:00080000:2.0:1789762596.747432:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762596.747436:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762596.747440:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762596.747449:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496151424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762596.747454:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762596.747494:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762596/real 1789762596] req@ffff8dd205265500 x1876699496151040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762596.747509:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496151040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762596.747516:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762596.747519:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762596.747522:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496151040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762596.747540:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496151168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762596.747557:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762596.747566:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762596/real 1789762596] req@ffff8dd205266680 x1876699496151168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762596.747574:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496151168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762596.747580:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762596.747582:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762596.747584:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496151168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762596.747588:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496151296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762596.747599:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762596.747605:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762596/real 1789762596] req@ffff8dd205267480 x1876699496151296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762596.747612:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496151296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762596.747617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762596.747619:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762596.747621:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496151296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762596.747624:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496151424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762596.747642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762596.747649:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762596/real 1789762596] req@ffff8dd205264a80 x1876699496151424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762596.747657:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496151424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762596.747662:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762596.747664:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762596.747666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496151424:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762597.323434:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762598.347321:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762599.371361:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762600.395320:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762601.419295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762601.867263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762601.867271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762601.867274:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762601.867285:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2581 00000100:00080000:2.0:1789762601.867295:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762601.867301:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762601.867333:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496151552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762601.867356:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762601.867360:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762601.867361:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762601.867365:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2581 00000100:00080000:2.0:1789762601.867369:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762601.867373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762601.867382:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762601.867397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496151680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762601.867406:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496151552:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762601.867407:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762601.867411:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762601.867412:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762601.867416:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2581 00000100:00080000:2.0:1789762601.867420:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762601.867423:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762601.867428:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762601.867438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496151808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762601.867447:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762601.867450:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762601.867452:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762601.867455:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2581 00000100:00080000:2.0:1789762601.867457:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762601.867461:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762601.867465:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762601.867473:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762601.867473:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496151936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762601.867512:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762601/real 1789762601] req@ffff8dd205267800 x1876699496151552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762601.867530:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496151552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762601.867538:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762601.867540:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762601.867543:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496151552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762601.867562:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496151680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762601.867581:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762601.867590:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762601/real 1789762601] req@ffff8dd205264380 x1876699496151680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762601.867598:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496151680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762601.867607:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762601.867609:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762601.867612:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496151680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762601.867615:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496151808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762601.867625:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762601.867631:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762601/real 1789762601] req@ffff8dd205264e00 x1876699496151808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762601.867638:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496151808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762601.867643:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762601.867645:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762601.867647:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496151808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762601.867650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496151936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762601.867660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762601.867665:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762601/real 1789762601] req@ffff8dd205267100 x1876699496151936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762601.867673:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496151936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762601.867677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762601.867679:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762601.867681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496151936:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762602.443293:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762603.467295:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762604.491332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762605.515355:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762606.539387:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762606.987310:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762606.987321:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762606.987326:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762606.987337:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2586 00000100:00080000:2.0:1789762606.987351:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762606.987359:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762606.987403:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496152064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762606.987433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762606.987436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762606.987437:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762606.987441:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2586 00000100:00080000:2.0:1789762606.987443:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762606.987447:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762606.987456:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762606.987465:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496152192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762606.987474:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762606.987477:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762606.987478:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762606.987481:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2586 00000100:00080000:2.0:1789762606.987483:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762606.987485:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762606.987489:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762606.987497:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496152320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762606.987503:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496152064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762606.987508:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762606.987513:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762606.987514:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762606.987516:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2586 00000100:00080000:2.0:1789762606.987518:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762606.987521:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762606.987524:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762606.987533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496152448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762606.987582:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762606.987626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762606/real 1789762606] req@ffff8dd205267b80 x1876699496152064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762606.987642:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496152064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762606.987670:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762606.987676:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762606.987680:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496152064:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762606.987689:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496152192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762606.987715:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762606.987727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762606/real 1789762606] req@ffff8dd205264000 x1876699496152192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762606.987736:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496152192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762606.987750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762606.987753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762606.987755:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496152192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762606.987781:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496152320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762606.987796:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762606.987804:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762606/real 1789762606] req@ffff8dd205265880 x1876699496152320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762606.987812:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496152320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762606.987817:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762606.987819:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762606.987821:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496152320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762606.987824:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496152448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762606.987835:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762606.987841:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762606/real 1789762606] req@ffff8dd205266a00 x1876699496152448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762606.987849:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496152448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762606.987854:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762606.987855:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762606.987857:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496152448:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762607.563301:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762608.587328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762609.611330:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762610.635327:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762611.659284:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762612.107285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762612.107300:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762612.107304:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762612.107312:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2591 00000100:00080000:3.0:1789762612.107322:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762612.107328:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762612.107361:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496152576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762612.107384:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762612.107386:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762612.107388:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762612.107391:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2591 00000100:00080000:3.0:1789762612.107393:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762612.107395:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762612.107402:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762612.107411:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496152704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762612.107418:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762612.107420:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762612.107421:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762612.107423:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2591 00000100:00080000:3.0:1789762612.107425:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762612.107427:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762612.107430:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762612.107437:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496152832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762612.107444:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762612.107447:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762612.107448:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762612.107451:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2591 00000100:00080000:3.0:1789762612.107453:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762612.107455:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762612.107458:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762612.107469:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496152576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762612.107469:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496152960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762612.107552:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496152704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762612.107571:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496152832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762612.107594:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496152960:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762612.683329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762613.707311:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762614.731347:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762615.179408:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789762615.179452:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762615.179468:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762615.179510:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762615.179520:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762615.179528:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789762615.179575:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762612/real 1789762615] req@ffff8dd203147480 x1876699496152960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762615.179615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496152960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762615.179627:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762615.179633:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762615.179637:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496152960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762615.179645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762612/real 1789762615] req@ffff8dd203145880 x1876699496152576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762615.179658:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496152576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762615.179678:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762615.179682:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762615.179685:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496152576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762615.179692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762612/real 1789762615] req@ffff8dd203147100 x1876699496152704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762615.179706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496152704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762615.179714:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762615.179718:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762615.179721:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496152704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762615.179728:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762612/real 1789762615] req@ffff8dd203144000 x1876699496152832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762615.179740:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496152832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762615.179748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762615.179752:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762615.179756:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496152832:192.168.203.109@tcp:8:kworker.0 00010000:00010000:0.0:1789762615.755307:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762616.779340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762617.227231:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762617.227242:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762617.227246:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762617.227258:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2596 00000100:00080000:2.0:1789762617.227269:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762617.227276:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762617.227307:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496153088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762617.227343:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762617.227347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762617.227349:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762617.227356:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2596 00000100:00080000:2.0:1789762617.227360:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762617.227366:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762617.227376:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496153088:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789762617.227377:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762617.227399:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496153216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762617.227412:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762617.227416:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762617.227418:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762617.227422:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2596 00000100:00080000:2.0:1789762617.227427:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762617.227431:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762617.227437:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789762617.227447:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762617.227451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496153344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762617.227465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762617.227469:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762617.227471:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762617.227475:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2596 00000100:00080000:2.0:1789762617.227480:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762617.227484:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762617.227491:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762617.227492:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762617/real 1789762617] req@ffff8dd205264700 x1876699496153088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762617.227507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496153472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762617.227519:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496153088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762617.227538:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762617.227543:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762617.227548:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496153088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762617.227590:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496153216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762617.227618:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762617.227636:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762617/real 1789762617] req@ffff8dd205266300 x1876699496153216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762617.227649:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496153216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762617.227659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762617.227663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762617.227667:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496153216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762617.227673:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496153344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762617.227696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762617.227709:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762617/real 1789762617] req@ffff8dd205265180 x1876699496153344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762617.227722:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496153344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762617.227731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762617.227734:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762617.227737:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496153344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762617.227744:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496153472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762617.227765:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762617.227778:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762617/real 1789762617] req@ffff8dd205266d80 x1876699496153472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762617.227791:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496153472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762617.227799:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762617.227802:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762617.227806:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496153472:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762617.803372:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762618.827393:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762619.851364:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762620.875388:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762621.899429:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762622.347302:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762622.347311:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762622.347316:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762622.347326:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2601 00000100:00080000:3.0:1789762622.347336:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762622.347343:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762622.347376:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496153600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762622.347413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762622.347417:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762622.347419:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762622.347423:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2601 00000100:00080000:3.0:1789762622.347427:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762622.347430:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762622.347440:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762622.347453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496153728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762622.347463:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762622.347468:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762622.347471:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496153600:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762622.347471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762622.347481:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2601 00000100:00080000:3.0:1789762622.347485:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762622.347489:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762622.347494:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762622.347507:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496153856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762622.347518:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762622.347522:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762622.347524:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762622.347528:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2601 00000100:00080000:3.0:1789762622.347532:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762622.347552:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762622.347553:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762622.347561:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762622.347582:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496153984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762622.347608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762622/real 1789762622] req@ffff8dd203145c00 x1876699496153600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762622.347628:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496153600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762622.347637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762622.347641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762622.347645:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496153600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762622.347667:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496153728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762622.347690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762622.347712:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762622/real 1789762622] req@ffff8dd203145180 x1876699496153728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762622.347723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496153728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762622.347731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762622.347734:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762622.347737:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496153728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762622.347742:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496153856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762622.347758:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762622.347778:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762622/real 1789762622] req@ffff8dd203147800 x1876699496153856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762622.347788:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496153856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762622.347796:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762622.347799:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762622.347802:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496153856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762622.347807:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496153984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762622.347821:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762622.347831:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762622/real 1789762622] req@ffff8dd203146300 x1876699496153984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762622.347842:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496153984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762622.347850:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762622.347854:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762622.347857:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496153984:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762622.923302:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762623.947294:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762624.971339:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762625.995286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762627.019309:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762627.467279:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762627.467287:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762627.467290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762627.467299:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2607 00000100:00080000:2.0:1789762627.467308:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762627.467313:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762627.467343:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496154112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762627.467366:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762627.467368:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762627.467370:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762627.467372:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2607 00000100:00080000:2.0:1789762627.467374:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762627.467377:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762627.467384:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762627.467395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496154240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762627.467413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762627.467415:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762627.467416:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762627.467419:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2607 00000100:00080000:2.0:1789762627.467421:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762627.467425:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762627.467428:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762627.467434:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496154368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762627.467441:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762627.467444:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762627.467445:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762627.467448:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2607 00000100:00080000:2.0:1789762627.467449:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762627.467453:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762627.467456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762627.467470:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496154496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762627.467471:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496154112:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1789762627.467595:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762627.467647:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762627/real 1789762627] req@ffff8dd205265f80 x1876699496154112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762627.467662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496154112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762627.467673:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762627.467676:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762627.467679:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496154112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762627.467685:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496154240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762627.467702:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762627.467711:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762627/real 1789762627] req@ffff8dd205266d80 x1876699496154240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762627.467719:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496154240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762627.467725:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762627.467727:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762627.467729:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496154240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762627.467733:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496154368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762627.467745:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762627.467751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762627/real 1789762627] req@ffff8dd205265180 x1876699496154368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762627.467758:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496154368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762627.467763:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762627.467765:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762627.467767:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496154368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762627.467798:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496154496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762627.467809:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762627.467816:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762627/real 1789762627] req@ffff8dd205266300 x1876699496154496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762627.467824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496154496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762627.467829:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762627.467830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762627.467832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496154496:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762628.043374:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762629.067357:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762630.091334:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762631.115293:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762632.139294:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762632.587259:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762632.587267:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762632.587272:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762632.587282:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2612 00000100:00080000:3.0:1789762632.587293:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762632.587299:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762632.587336:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496154624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762632.587363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762632.587366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762632.587367:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762632.587371:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2612 00000100:00080000:3.0:1789762632.587374:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762632.587379:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762632.587390:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762632.587405:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496154624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762632.587408:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496154752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762632.587433:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762632.587436:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762632.587438:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762632.587441:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2612 00000100:00080000:3.0:1789762632.587444:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762632.587448:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762632.587453:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789762632.587475:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762632.587480:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496154880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762632.587489:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762632.587492:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762632.587493:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762632.587496:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2612 00000100:00080000:3.0:1789762632.587499:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762632.587502:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762632.587505:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762632.587514:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762632/real 1789762632] req@ffff8dd203144a80 x1876699496154624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762632.587516:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496155008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762632.587550:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496154624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762632.587560:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762632.587563:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762632.587566:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496154624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762632.587583:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496154752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762632.587599:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762632.587608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762632/real 1789762632] req@ffff8dd203144700 x1876699496154752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762632.587616:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496154752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762632.587622:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762632.587624:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762632.587626:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496154752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762632.587630:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496154880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762632.587641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762632.587646:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762632/real 1789762632] req@ffff8dd203145f80 x1876699496154880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762632.587654:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496154880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762632.587659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762632.587661:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762632.587663:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496154880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762632.587666:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496155008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762632.587676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762632.587682:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762632/real 1789762632] req@ffff8dd203144380 x1876699496155008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762632.587689:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496155008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762632.587695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762632.587697:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762632.587699:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496155008:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762633.163332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762634.187291:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762635.211312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762636.235425:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762637.259355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762637.707292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762637.707302:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762637.707307:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762637.707318:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2617 00000100:00080000:0.0:1789762637.707330:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762637.707337:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762637.707390:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496155136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762637.707439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762637.707442:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762637.707444:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762637.707448:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2617 00000100:00080000:0.0:1789762637.707452:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762637.707457:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762637.707469:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762637.707485:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496155264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762637.707497:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496155136:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762637.707507:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762637.707513:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762637.707515:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762637.707523:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2617 00000100:00080000:0.0:1789762637.707528:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762637.707532:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762637.707539:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762637.707553:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496155392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762637.707561:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762637.707566:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762637.707567:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762637.707575:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789762637.707579:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2617 00000100:00080000:0.0:1789762637.707583:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762637.707587:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762637.707593:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762637.707603:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496155520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762637.707632:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762637/real 1789762637] req@ffff8dd211835f80 x1876699496155136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762637.707647:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496155136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762637.707656:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762637.707659:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762637.707662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496155136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762637.707682:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496155264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762637.707698:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762637.707706:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762637/real 1789762637] req@ffff8dd211834380 x1876699496155264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762637.707714:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496155264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762637.707719:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762637.707722:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762637.707724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496155264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762637.707729:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496155392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762637.707740:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762637.707746:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762637/real 1789762637] req@ffff8dd211836680 x1876699496155392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762637.707753:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496155392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762637.707759:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762637.707761:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762637.707762:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496155392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762637.707766:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496155520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762637.707776:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762637.707782:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762637/real 1789762637] req@ffff8dd211834a80 x1876699496155520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762637.707789:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496155520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762637.707794:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762637.707796:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762637.707798:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496155520:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762638.283305:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762639.307289:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762640.331316:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762641.355286:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762642.379313:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762642.827288:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762642.827298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762642.827303:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762642.827313:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2622 00000100:00080000:1.0:1789762642.827331:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762642.827339:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762642.827395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496155648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762642.827432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762642.827436:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762642.827438:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762642.827442:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2622 00000100:00080000:1.0:1789762642.827447:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762642.827451:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762642.827461:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762642.827469:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496155648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762642.827477:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496155776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762642.827490:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762642.827494:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762642.827496:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762642.827501:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2622 00000100:00080000:1.0:1789762642.827517:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762642.827521:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789762642.827526:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789762642.827527:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762642.827542:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496155904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762642.827553:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762642.827557:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762642.827560:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762642.827563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762642/real 1789762642] req@ffff8dd20504df80 x1876699496155648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762642.827567:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2622 00000100:00080000:1.0:1789762642.827574:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762642.827580:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762642.827586:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496155648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762642.827586:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762642.827598:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789762642.827601:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496156032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762642.827603:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762642.827611:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496155648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762642.827656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496155776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762642.827681:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762642.827692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762642/real 1789762642] req@ffff8dd20504fb80 x1876699496155776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762642.827702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496155776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762642.827710:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762642.827713:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762642.827716:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496155776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762642.827723:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496155904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762642.827755:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762642.827766:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762642/real 1789762642] req@ffff8dd20504f800 x1876699496155904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762642.827780:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496155904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762642.827786:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762642.827788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762642.827790:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496155904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762642.827794:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496156032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762642.827807:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762642.827813:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762642/real 1789762642] req@ffff8dd20504c700 x1876699496156032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762642.827821:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496156032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762642.827826:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762642.827828:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762642.827830:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496156032:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762643.403334:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762644.427297:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762645.451305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762646.475302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762647.499379:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762647.947318:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762647.947327:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762647.947330:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762647.947339:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2627 00000100:00080000:2.0:1789762647.947362:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762647.947369:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762647.947403:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496156160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762647.947426:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762647.947429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762647.947430:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762647.947433:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2627 00000100:00080000:2.0:1789762647.947435:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762647.947437:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762647.947444:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762647.947453:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496156288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762647.947461:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762647.947463:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762647.947465:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762647.947467:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2627 00000100:00080000:2.0:1789762647.947469:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762647.947471:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762647.947475:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762647.947479:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496156160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762647.947485:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496156416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762647.947493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762647.947495:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762647.947497:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762647.947499:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2627 00000100:00080000:2.0:1789762647.947501:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762647.947504:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762647.947507:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762647.947515:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496156544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762647.947540:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762647.947574:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762647/real 1789762647] req@ffff8dd205264700 x1876699496156160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762647.947588:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496156160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762647.947600:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762647.947603:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762647.947606:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496156160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762647.947611:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496156288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762647.947625:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762647.947632:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762647/real 1789762647] req@ffff8dd205266a00 x1876699496156288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762647.947640:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496156288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762647.947646:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762647.947648:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762647.947650:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496156288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762647.947668:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496156416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762647.947679:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762647.947685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762647/real 1789762647] req@ffff8dd205265880 x1876699496156416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762647.947693:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496156416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762647.947699:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762647.947701:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762647.947703:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496156416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762647.947707:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496156544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762647.947716:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762647.947722:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762647/real 1789762647] req@ffff8dd205264000 x1876699496156544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762647.947730:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496156544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762647.947735:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762647.947736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762647.947738:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496156544:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762648.523319:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762649.547358:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762650.571342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762651.595291:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762652.619350:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762653.067256:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762653.067266:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762653.067271:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762653.067282:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2632 00000100:00080000:1.0:1789762653.067293:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762653.067300:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762653.067350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496156672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762653.067373:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762653.067375:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762653.067377:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762653.067380:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2632 00000100:00080000:1.0:1789762653.067384:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762653.067387:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762653.067401:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762653.067402:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496156672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762653.067415:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496156800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762653.067424:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762653.067427:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762653.067428:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762653.067431:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2632 00000100:00080000:1.0:1789762653.067433:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762653.067436:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762653.067439:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762653.067447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496156928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762653.067454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762653.067457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762653.067458:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762653.067460:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2632 00000100:00080000:1.0:1789762653.067462:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762653.067466:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762653.067469:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762653.067473:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762653.067479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496157056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762653.067511:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762653/real 1789762653] req@ffff8dd20504ce00 x1876699496156672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762653.067525:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496156672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762653.067533:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762653.067536:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762653.067539:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496156672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762653.067557:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496156800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762653.067574:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762653.067582:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762653/real 1789762653] req@ffff8dd20504f100 x1876699496156800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762653.067591:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496156800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762653.067605:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762653.067608:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762653.067610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496156800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762653.067614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496156928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762653.067627:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762653.067634:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762653/real 1789762653] req@ffff8dd20504ed80 x1876699496156928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762653.067642:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496156928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762653.067647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762653.067648:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762653.067650:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496156928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762653.067660:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496157056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762653.067671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762653.067677:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762653/real 1789762653] req@ffff8dd20504d880 x1876699496157056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762653.067684:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496157056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762653.067694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762653.067696:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762653.067698:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496157056:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762653.643323:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762654.667327:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762655.691319:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762656.715335:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762657.739306:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762658.187252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762658.187260:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762658.187264:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762658.187280:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2637 00000100:00080000:0.0:1789762658.187289:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762658.187295:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762658.187326:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496157184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762658.187349:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762658.187352:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762658.187353:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762658.187356:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2637 00000100:00080000:0.0:1789762658.187359:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762658.187361:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762658.187369:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762658.187380:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496157184:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762658.187381:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496157312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762658.187395:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762658.187397:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762658.187399:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762658.187402:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2637 00000100:00080000:0.0:1789762658.187404:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762658.187406:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762658.187410:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762658.187419:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496157440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762658.187426:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762658.187428:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762658.187430:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762658.187432:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2637 00000100:00080000:0.0:1789762658.187435:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762658.187440:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789762658.187440:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789762658.187447:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762658.187457:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496157568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762658.187480:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762658/real 1789762658] req@ffff8dd211836d80 x1876699496157184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762658.187494:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496157184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762658.187502:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762658.187505:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762658.187508:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496157184:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762658.187528:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496157312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762658.187543:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762658.187551:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762658/real 1789762658] req@ffff8dd211835180 x1876699496157312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762658.187559:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496157312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762658.187565:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762658.187567:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762658.187569:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496157312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762658.187573:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496157440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762658.187585:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762658.187591:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762658/real 1789762658] req@ffff8dd211837480 x1876699496157440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762658.187599:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496157440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762658.187604:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762658.187606:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762658.187608:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496157440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762658.187611:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496157568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762658.187621:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762658.187627:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762658/real 1789762658] req@ffff8dd211834e00 x1876699496157568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762658.187634:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496157568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762658.187639:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762658.187641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762658.187642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496157568:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762658.763462:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762659.787292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762660.811379:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762661.835314:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762662.859336:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762663.307254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762663.307265:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762663.307269:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762663.307279:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2642 00000100:00080000:0.0:1789762663.307307:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762663.307314:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762663.307362:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496157696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762663.307397:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762663.307400:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762663.307403:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762663.307407:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2642 00000100:00080000:0.0:1789762663.307410:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762663.307414:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762663.307422:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762663.307444:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496157824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762663.307453:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496157696:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762663.307457:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762663.307462:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762663.307464:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762663.307467:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2642 00000100:00080000:0.0:1789762663.307470:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762663.307473:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762663.307477:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762663.307486:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496157952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762663.307494:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762663.307497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762663.307498:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762663.307500:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2642 00000100:00080000:0.0:1789762663.307511:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762663.307513:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762663.307517:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762663.307522:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762663.307529:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496158080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762663.307560:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762663/real 1789762663] req@ffff8dd211834e00 x1876699496157696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762663.307578:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496157696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762663.307586:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762663.307589:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762663.307592:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496157696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762663.307612:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496157824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762663.307634:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762663.307648:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762663/real 1789762663] req@ffff8dd211837480 x1876699496157824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762663.307681:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496157824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762663.307687:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762663.307689:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762663.307692:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496157824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762663.307696:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496157952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762663.307712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762663.307720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762663/real 1789762663] req@ffff8dd211835180 x1876699496157952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762663.307727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496157952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762663.307737:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762663.307739:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762663.307742:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496157952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762663.307745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496158080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762663.307755:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762663.307760:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762663/real 1789762663] req@ffff8dd211836d80 x1876699496158080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762663.307767:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496158080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762663.307773:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762663.307774:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762663.307776:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496158080:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762663.883307:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762664.907322:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762665.931315:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762666.955291:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762667.979325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762668.427258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762668.427266:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762668.427270:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762668.427278:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2648 00000100:00080000:2.0:1789762668.427287:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762668.427293:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762668.427331:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496158208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762668.427351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762668.427354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762668.427355:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762668.427358:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2648 00000100:00080000:2.0:1789762668.427360:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762668.427363:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762668.427370:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762668.427379:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496158336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762668.427388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762668.427390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762668.427392:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762668.427394:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2648 00000100:00080000:2.0:1789762668.427396:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762668.427402:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496158208:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762668.427402:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762668.427409:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762668.427417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496158464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762668.427424:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762668.427426:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762668.427428:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762668.427431:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2648 00000100:00080000:2.0:1789762668.427433:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762668.427436:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762668.427439:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762668.427447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496158592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762668.427460:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762668.427494:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762668/real 1789762668] req@ffff8dd205267b80 x1876699496158208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762668.427508:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496158208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762668.427515:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762668.427519:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762668.427522:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496158208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762668.427526:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496158336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762668.427540:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762668.427549:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762668/real 1789762668] req@ffff8dd205267100 x1876699496158336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762668.427556:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496158336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762668.427561:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762668.427563:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762668.427565:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496158336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762668.427583:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496158464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762668.427595:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762668.427601:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762668/real 1789762668] req@ffff8dd205264e00 x1876699496158464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762668.427609:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496158464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762668.427614:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762668.427616:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762668.427618:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496158464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762668.427621:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496158592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762668.427631:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762668.427637:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762668/real 1789762668] req@ffff8dd205264380 x1876699496158592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762668.427644:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496158592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762668.427649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762668.427651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762668.427653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496158592:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762669.003291:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762670.027299:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762671.051313:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762672.075294:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762673.099300:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762673.547297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762673.547306:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762673.547309:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762673.547318:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2653 00000100:00080000:3.0:1789762673.547327:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762673.547333:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762673.547368:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496158720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762673.547401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762673.547403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762673.547405:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762673.547408:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2653 00000100:00080000:3.0:1789762673.547410:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762673.547412:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762673.547419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762673.547428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496158848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762673.547435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762673.547438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762673.547439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762673.547442:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2653 00000100:00080000:3.0:1789762673.547444:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762673.547450:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496158720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762673.547450:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762673.547455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762673.547463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496158976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762673.547472:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762673.547475:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762673.547476:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762673.547478:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2653 00000100:00080000:3.0:1789762673.547481:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762673.547483:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762673.547486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762673.547493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496159104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762673.547529:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762673.547563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762673/real 1789762673] req@ffff8dd203145500 x1876699496158720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762673.547577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496158720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762673.547594:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762673.547597:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762673.547601:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496158720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762673.547606:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496158848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762673.547624:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762673.547636:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762673/real 1789762673] req@ffff8dd203146a00 x1876699496158848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762673.547657:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496158848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762673.547683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762673.547686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762673.547688:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496158848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762673.547708:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496158976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762673.547725:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762673.547733:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762673/real 1789762673] req@ffff8dd203144e00 x1876699496158976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762673.547755:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496158976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762673.547764:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762673.547767:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762673.547770:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496158976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762673.547776:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496159104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762673.547796:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762673.547808:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762673/real 1789762673] req@ffff8dd203146d80 x1876699496159104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762673.547817:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496159104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762673.547828:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762673.547830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762673.547833:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496159104:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762674.123293:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762675.147294:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762676.171295:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762677.195363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762678.219288:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762678.667244:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762678.667254:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762678.667258:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762678.667269:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2658 00000100:00080000:3.0:1789762678.667281:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762678.667289:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762678.667323:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496159232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762678.667347:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762678.667350:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762678.667351:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762678.667354:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2658 00000100:00080000:3.0:1789762678.667357:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762678.667360:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762678.667367:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762678.667377:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496159360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762678.667385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762678.667388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762678.667389:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762678.667392:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2658 00000100:00080000:3.0:1789762678.667395:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762678.667397:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762678.667401:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762678.667405:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496159232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762678.667410:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496159488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762678.667420:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762678.667422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762678.667423:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762678.667425:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2658 00000100:00080000:3.0:1789762678.667427:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762678.667429:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762678.667431:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762678.667446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496159616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762678.667496:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496159360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762678.667533:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496159488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762678.667557:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496159616:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762679.243331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762680.267307:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762681.291314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762681.739419:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789762681.739461:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762681.739475:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762681.739506:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762681.739513:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789762681.739519:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789762681.739553:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762678/real 1789762681] req@ffff8dd203144e00 x1876699496159488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762681.739575:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496159488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762681.739584:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762681.739588:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762681.739591:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496159488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762681.739596:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762678/real 1789762681] req@ffff8dd203146a00 x1876699496159616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762681.739604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496159616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762681.739614:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762681.739616:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762681.739618:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496159616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762681.739622:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762678/real 1789762681] req@ffff8dd203147b80 x1876699496159232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762681.739629:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496159232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762681.739634:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762681.739636:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762681.739638:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496159232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762681.739643:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762678/real 1789762681] req@ffff8dd203146d80 x1876699496159360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762681.739655:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496159360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762681.739663:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762681.739666:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762681.739669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496159360:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762682.315286:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762683.339332:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762683.787236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762683.787247:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762683.787252:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762683.787263:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2663 00000100:00080000:0.0:1789762683.787276:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762683.787283:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762683.787325:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496159744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762683.787349:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762683.787353:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762683.787355:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762683.787358:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2663 00000100:00080000:0.0:1789762683.787361:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762683.787363:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762683.787373:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762683.787383:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496159872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762683.787392:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762683.787395:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762683.787395:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496159744:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762683.787399:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762683.787405:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2663 00000100:00080000:0.0:1789762683.787408:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762683.787410:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762683.787414:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762683.787423:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496160000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762683.787430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762683.787432:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762683.787433:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762683.787435:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2663 00000100:00080000:0.0:1789762683.787438:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762683.787441:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762683.787444:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762683.787450:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496160128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762683.787484:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762683.787538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762683/real 1789762683] req@ffff8dd211834a80 x1876699496159744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762683.787561:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496159744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762683.787573:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762683.787578:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762683.787583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496159744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762683.787610:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496159872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762683.787636:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762683.787650:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762683/real 1789762683] req@ffff8dd211836680 x1876699496159872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762683.787663:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496159872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762683.787680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762683.787684:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762683.787687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496159872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762683.787694:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496160000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762683.787718:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762683.787730:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762683/real 1789762683] req@ffff8dd211834380 x1876699496160000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762683.787743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496160000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762683.787757:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762683.787761:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762683.787765:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496160000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762683.787770:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496160128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762683.787789:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762683.787800:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762683/real 1789762683] req@ffff8dd211835f80 x1876699496160128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762683.787811:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496160128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762683.787819:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762683.787822:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762683.787826:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496160128:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762684.363311:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762685.387333:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762686.411340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762687.435393:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762688.459292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762688.907235:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762688.907243:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762688.907247:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762688.907255:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2668 00000100:00080000:3.0:1789762688.907264:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762688.907269:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762688.907298:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496160256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762688.907330:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762688.907333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762688.907334:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762688.907337:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2668 00000100:00080000:3.0:1789762688.907340:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762688.907342:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762688.907349:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762688.907358:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496160384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762688.907366:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762688.907370:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496160256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762688.907371:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762688.907375:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762688.907381:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2668 00000100:00080000:3.0:1789762688.907384:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762688.907386:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762688.907389:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762688.907399:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496160512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762688.907406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762688.907409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762688.907411:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762688.907413:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2668 00000100:00080000:3.0:1789762688.907415:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762688.907418:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762688.907421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762688.907429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496160640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762688.907432:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762688.907475:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762688/real 1789762688] req@ffff8dd203145500 x1876699496160256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762688.907490:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496160256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762688.907502:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762688.907505:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762688.907508:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496160256:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762688.907529:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496160384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762688.907554:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762688.907568:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762688/real 1789762688] req@ffff8dd203144380 x1876699496160384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762688.907581:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496160384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762688.907587:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762688.907589:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762688.907591:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496160384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762688.907596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496160512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762688.907611:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762688.907618:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762688/real 1789762688] req@ffff8dd203145f80 x1876699496160512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762688.907626:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496160512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762688.907632:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762688.907634:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762688.907636:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496160512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762688.907640:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496160640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762688.907650:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762688.907665:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762688/real 1789762688] req@ffff8dd203144700 x1876699496160640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762688.907673:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496160640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762688.907678:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762688.907680:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762688.907681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496160640:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762689.483324:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762690.507390:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762691.531290:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762692.555330:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762693.579323:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762694.027340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762694.027348:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762694.027351:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762694.027359:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2673 00000100:00080000:0.0:1789762694.027369:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762694.027374:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762694.027408:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496160768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762694.027432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762694.027435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762694.027436:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762694.027439:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2673 00000100:00080000:0.0:1789762694.027441:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762694.027443:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762694.027450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762694.027458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496160896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762694.027464:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762694.027466:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762694.027467:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762694.027470:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2673 00000100:00080000:0.0:1789762694.027471:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762694.027473:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762694.027476:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762694.027482:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496161024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762694.027489:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762694.027492:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762694.027494:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762694.027497:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2673 00000100:00080000:0.0:1789762694.027500:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762694.027503:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762694.027507:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762694.027515:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496161152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762694.027529:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496160768:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1789762694.027618:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762694.027665:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762694/real 1789762694] req@ffff8dd211834000 x1876699496160768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762694.027681:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496160768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762694.027690:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762694.027694:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762694.027697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496160768:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762694.027703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496160896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762694.027721:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762694.027731:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762694/real 1789762694] req@ffff8dd211836a00 x1876699496160896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762694.027745:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496160896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762694.027754:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762694.027757:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762694.027761:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496160896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762694.027768:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496161024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762694.027790:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762694.027801:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762694/real 1789762694] req@ffff8dd211835c00 x1876699496161024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762694.027809:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496161024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762694.027815:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762694.027817:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762694.027819:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496161024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762694.027848:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496161152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762694.027862:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762694.027868:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762694/real 1789762694] req@ffff8dd211834700 x1876699496161152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762694.027881:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496161152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762694.027886:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762694.027888:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762694.027890:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496161152:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762694.603287:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762695.627330:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762696.651377:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762697.675413:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762698.699304:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762699.147320:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762699.147333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762699.147338:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762699.147352:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2678 00000100:00080000:3.0:1789762699.147374:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762699.147383:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762699.147436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496161280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762699.147486:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762699.147490:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762699.147491:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762699.147495:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2678 00000100:00080000:3.0:1789762699.147498:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762699.147501:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762699.147512:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762699.147542:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496161408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762699.147554:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762699.147559:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496161280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762699.147560:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762699.147564:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762699.147571:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2678 00000100:00080000:3.0:1789762699.147575:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762699.147580:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762699.147587:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762699.147602:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496161536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762699.147614:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762699.147618:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762699.147620:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762699.147625:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2678 00000100:00080000:3.0:1789762699.147630:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762699.147634:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762699.147638:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762699.147644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762699.147650:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496161664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762699.147696:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762699/real 1789762699] req@ffff8dd203144a80 x1876699496161280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762699.147712:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496161280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762699.147721:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762699.147724:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762699.147727:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496161280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762699.147749:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496161408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762699.147767:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762699.147776:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762699/real 1789762699] req@ffff8dd203146300 x1876699496161408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762699.147790:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496161408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762699.147796:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762699.147798:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762699.147800:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496161408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762699.147803:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496161536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762699.147816:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762699.147822:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762699/real 1789762699] req@ffff8dd203147800 x1876699496161536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762699.147830:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496161536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762699.147835:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762699.147837:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762699.147839:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496161536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762699.147842:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496161664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762699.147853:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762699.147858:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762699/real 1789762699] req@ffff8dd203145180 x1876699496161664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762699.147866:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496161664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762699.147876:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762699.147878:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762699.147880:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496161664:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762699.723330:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762700.747352:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762701.771304:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762702.795319:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762703.819318:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762704.267269:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762704.267282:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762704.267287:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762704.267296:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2683 00000100:00080000:3.0:1789762704.267306:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762704.267312:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762704.267352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496161792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762704.267382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762704.267385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762704.267390:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762704.267394:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2683 00000100:00080000:3.0:1789762704.267397:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762704.267402:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762704.267412:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762704.267428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496161920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762704.267432:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496161792:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762704.267449:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762704.267452:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762704.267454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762704.267458:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2683 00000100:00080000:3.0:1789762704.267461:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762704.267464:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762704.267469:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762704.267478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496162048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762704.267491:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762704.267493:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762704.267495:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762704.267498:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2683 00000100:00080000:3.0:1789762704.267500:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762704.267502:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762704.267505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762704.267508:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762704.267515:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496162176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762704.267563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762704/real 1789762704] req@ffff8dd203145c00 x1876699496161792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762704.267585:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496161792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762704.267598:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762704.267603:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762704.267606:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496161792:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762704.267626:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496161920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762704.267646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762704.267655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762704/real 1789762704] req@ffff8dd203144000 x1876699496161920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762704.267664:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496161920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762704.267669:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762704.267672:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762704.267674:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496161920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762704.267679:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496162048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762704.267699:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762704.267711:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762704/real 1789762704] req@ffff8dd203147100 x1876699496162048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762704.267724:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496162048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762704.267733:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762704.267736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762704.267739:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496162048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762704.267743:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496162176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762704.267758:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762704.267765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762704/real 1789762704] req@ffff8dd203145880 x1876699496162176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762704.267773:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496162176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762704.267778:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762704.267780:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762704.267783:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496162176:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762704.843350:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762705.867304:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762706.891340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762707.915368:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762708.939366:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762709.387296:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762709.387305:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762709.387308:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762709.387316:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2688 00000100:00080000:0.0:1789762709.387326:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762709.387339:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762709.387375:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496162304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762709.387397:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762709.387400:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762709.387402:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762709.387404:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2688 00000100:00080000:0.0:1789762709.387407:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762709.387409:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762709.387416:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762709.387426:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496162432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762709.387433:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762709.387436:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762709.387437:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762709.387440:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2688 00000100:00080000:0.0:1789762709.387443:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762709.387448:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496162304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762709.387449:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762709.387453:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762709.387462:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496162560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762709.387470:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762709.387472:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762709.387474:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762709.387476:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2688 00000100:00080000:0.0:1789762709.387480:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762709.387482:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762709.387485:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762709.387493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496162688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762709.387514:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762709.387554:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762709/real 1789762709] req@ffff8dd211837100 x1876699496162304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762709.387573:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496162304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762709.387584:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762709.387588:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762709.387592:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496162304:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762709.387600:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496162432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762709.387621:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762709.387633:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762709/real 1789762709] req@ffff8dd211836300 x1876699496162432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762709.387645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496162432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762709.387654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762709.387658:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762709.387661:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496162432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762709.387687:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496162560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762709.387705:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762709.387716:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762709/real 1789762709] req@ffff8dd211837800 x1876699496162560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762709.387728:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496162560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762709.387736:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762709.387739:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762709.387743:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496162560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762709.387749:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496162688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762709.387764:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762709.387773:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762709/real 1789762709] req@ffff8dd211837b80 x1876699496162688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762709.387785:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496162688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762709.387794:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762709.387797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762709.387800:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496162688:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762709.963334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762710.987757:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762712.011299:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762713.035301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762714.059316:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762714.507254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762714.507268:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762714.507274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762714.507286:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2694 00000100:00080000:0.0:1789762714.507301:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762714.507311:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762714.507367:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496162816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762714.507401:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762714.507405:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762714.507407:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762714.507412:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2694 00000100:00080000:0.0:1789762714.507416:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762714.507420:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762714.507433:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762714.507448:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496162944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762714.507457:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496162816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762714.507465:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762714.507470:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762714.507472:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762714.507477:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2694 00000100:00080000:0.0:1789762714.507481:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762714.507486:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762714.507493:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762714.507507:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496163072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762714.507520:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762714.507524:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762714.507526:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762714.507531:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2694 00000100:00080000:0.0:1789762714.507535:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762714.507540:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789762714.507542:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762714.507550:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762714.507564:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496163200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762714.507592:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762714/real 1789762714] req@ffff8dd211837b80 x1876699496162816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762714.507608:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496162816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762714.507618:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762714.507621:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762714.507625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496162816:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762714.507650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496162944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762714.507669:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762714.507679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762714/real 1789762714] req@ffff8dd211837800 x1876699496162944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762714.507688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496162944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762714.507694:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762714.507697:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762714.507699:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496162944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762714.507702:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496163072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762714.507714:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762714.507720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762714/real 1789762714] req@ffff8dd211836300 x1876699496163072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762714.507728:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496163072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762714.507734:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762714.507736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762714.507737:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496163072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762714.507741:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496163200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762714.507752:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762714.507759:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762714/real 1789762714] req@ffff8dd211837100 x1876699496163200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762714.507767:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496163200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762714.507772:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762714.507774:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762714.507775:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496163200:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762715.083299:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762716.107319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762717.131317:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762718.155344:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762719.179308:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762719.627246:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762719.627256:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762719.627261:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762719.627272:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2699 00000100:00080000:0.0:1789762719.627284:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762719.627291:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762719.627329:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496163328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762719.627355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762719.627357:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762719.627359:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762719.627363:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2699 00000100:00080000:0.0:1789762719.627366:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762719.627370:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762719.627379:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762719.627388:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496163328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762719.627394:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496163456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762719.627403:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762719.627406:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762719.627408:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762719.627410:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2699 00000100:00080000:0.0:1789762719.627413:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762719.627415:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762719.627419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762719.627428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496163584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762719.627435:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762719.627437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762719.627438:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762719.627441:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2699 00000100:00080000:0.0:1789762719.627443:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762719.627446:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762719.627448:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762719.627453:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762719.627458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496163712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762719.627491:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762719/real 1789762719] req@ffff8dd211834700 x1876699496163328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762719.627505:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496163328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762719.627513:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762719.627515:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762719.627518:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496163328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762719.627536:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496163456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762719.627550:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762719.627558:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762719/real 1789762719] req@ffff8dd211835c00 x1876699496163456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762719.627565:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496163456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762719.627571:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762719.627573:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762719.627575:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496163456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762719.627579:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496163584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762719.627589:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762719.627595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762719/real 1789762719] req@ffff8dd211836a00 x1876699496163584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762719.627602:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496163584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762719.627607:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762719.627609:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762719.627611:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496163584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762719.627614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496163712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762719.627624:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762719.627629:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762719/real 1789762719] req@ffff8dd211834000 x1876699496163712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762719.627641:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496163712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762719.627646:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762719.627648:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762719.627650:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496163712:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762720.203286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762721.227423:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762722.251369:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762723.275329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762724.299403:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762724.747300:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762724.747309:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762724.747312:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762724.747321:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2704 00000100:00080000:1.0:1789762724.747330:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762724.747335:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762724.747367:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496163840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762724.747391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762724.747394:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762724.747395:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762724.747398:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2704 00000100:00080000:1.0:1789762724.747400:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762724.747403:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762724.747411:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762724.747420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496163968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762724.747427:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496163840:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762724.747430:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762724.747437:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762724.747439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762724.747442:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2704 00000100:00080000:1.0:1789762724.747444:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762724.747446:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762724.747449:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762724.747459:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496164096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762724.747465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762724.747468:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762724.747469:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762724.747471:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2704 00000100:00080000:1.0:1789762724.747486:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762724.747489:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762724.747492:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762724.747497:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762724.747503:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496164224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762724.747536:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762724/real 1789762724] req@ffff8dd20504c000 x1876699496163840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762724.747551:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496163840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762724.747564:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762724.747567:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762724.747569:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496163840:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762724.747589:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496163968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762724.747604:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762724.747612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762724/real 1789762724] req@ffff8dd20504ea00 x1876699496163968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762724.747620:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496163968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762724.747625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762724.747627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762724.747629:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496163968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762724.747633:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496164096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762724.747644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762724.747649:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762724/real 1789762724] req@ffff8dd20504c380 x1876699496164096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762724.747656:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496164096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762724.747662:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762724.747663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762724.747665:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496164096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762724.747669:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496164224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762724.747679:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762724.747684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762724/real 1789762724] req@ffff8dd20504dc00 x1876699496164224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762724.747692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496164224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762724.747697:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762724.747698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762724.747700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496164224:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762725.323334:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762726.347306:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762727.371316:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762728.395284:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762729.419337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762729.867278:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762729.867286:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762729.867291:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762729.867300:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2709 00000100:00080000:0.0:1789762729.867310:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762729.867317:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762729.867354:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496164352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762729.867381:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762729.867386:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762729.867388:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762729.867392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2709 00000100:00080000:0.0:1789762729.867397:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762729.867402:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762729.867412:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762729.867424:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496164352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762729.867427:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496164480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762729.867443:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762729.867446:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762729.867448:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762729.867453:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2709 00000100:00080000:0.0:1789762729.867457:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762729.867460:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762729.867466:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762729.867477:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496164608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762729.867488:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762729.867491:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789762729.867496:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762729.867500:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762729.867506:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2709 00000100:00080000:0.0:1789762729.867513:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762729.867518:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762729.867524:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762729.867532:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762729/real 1789762729] req@ffff8dd211835f80 x1876699496164352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762729.867539:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496164736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762729.867551:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496164352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762729.867562:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762729.867565:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762729.867568:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496164352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762729.867588:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496164480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762729.867603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762729.867611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762729/real 1789762729] req@ffff8dd211834380 x1876699496164480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762729.867619:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496164480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762729.867625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762729.867627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762729.867629:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496164480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762729.867632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496164608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762729.867645:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762729.867655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762729/real 1789762729] req@ffff8dd211836680 x1876699496164608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762729.867670:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496164608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762729.867675:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762729.867677:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762729.867679:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496164608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762729.867683:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496164736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762729.867696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762729.867702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762729/real 1789762729] req@ffff8dd211834a80 x1876699496164736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762729.867710:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496164736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762729.867714:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762729.867716:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762729.867718:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496164736:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762730.443331:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762731.467383:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762732.491344:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762733.515361:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762734.539308:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762734.987251:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762734.987260:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762734.987264:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762734.987272:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2714 00000100:00080000:0.0:1789762734.987282:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762734.987288:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762734.987320:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496164864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762734.987347:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762734.987351:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762734.987353:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762734.987358:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2714 00000100:00080000:0.0:1789762734.987362:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762734.987366:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762734.987376:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762734.987389:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496164992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762734.987397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762734.987400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762734.987402:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762734.987405:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2714 00000100:00080000:0.0:1789762734.987407:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762734.987411:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762734.987415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762734.987422:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496164864:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762734.987430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496165120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762734.987444:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762734.987448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762734.987451:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762734.987454:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2714 00000100:00080000:0.0:1789762734.987475:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762734.987479:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762734.987484:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762734.987486:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762734.987496:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496165248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762734.987534:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762734/real 1789762734] req@ffff8dd211836d80 x1876699496164864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762734.987549:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496164864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762734.987557:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762734.987561:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762734.987564:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496164864:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762734.987569:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496164992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762734.987586:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762734.987595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762734/real 1789762734] req@ffff8dd211835180 x1876699496164992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762734.987603:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496164992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762734.987608:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762734.987611:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762734.987613:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496164992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762734.987631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496165120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762734.987643:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762734.987648:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762734/real 1789762734] req@ffff8dd211837480 x1876699496165120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762734.987656:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496165120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762734.987662:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762734.987663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762734.987665:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496165120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762734.987669:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496165248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762734.987679:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762734.987685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762734/real 1789762734] req@ffff8dd211834e00 x1876699496165248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762734.987708:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496165248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762734.987714:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762734.987716:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762734.987718:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496165248:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762735.563334:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762736.587310:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762737.611322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762738.635344:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762739.659371:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762740.107249:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762740.107256:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762740.107260:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762740.107268:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2719 00000100:00080000:0.0:1789762740.107277:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762740.107283:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762740.107354:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496165376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762740.107374:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762740.107377:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762740.107379:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762740.107383:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2719 00000100:00080000:0.0:1789762740.107386:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762740.107390:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762740.107413:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762740.107429:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496165504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762740.107439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762740.107441:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762740.107442:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762740.107445:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2719 00000100:00080000:0.0:1789762740.107448:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762740.107451:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762740.107456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762740.107459:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496165376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762740.107466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496165632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762740.107475:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762740.107477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762740.107478:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762740.107481:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2719 00000100:00080000:0.0:1789762740.107483:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762740.107485:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762740.107487:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762740.107495:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496165760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762740.107534:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762740.107582:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762740/real 1789762740] req@ffff8dd211834e00 x1876699496165376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762740.107598:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496165376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762740.107609:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762740.107624:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762740.107628:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496165376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762740.107636:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496165504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762740.107676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762740.107690:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762740/real 1789762740] req@ffff8dd211837480 x1876699496165504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762740.107699:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496165504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762740.107715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762740.107718:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762740.107720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496165504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762740.107740:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496165632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762740.107754:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762740.107761:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762740/real 1789762740] req@ffff8dd211835180 x1876699496165632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762740.107769:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496165632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762740.107774:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762740.107776:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762740.107778:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496165632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762740.107781:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496165760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762740.107791:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762740.107796:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762740/real 1789762740] req@ffff8dd211836d80 x1876699496165760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762740.107803:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496165760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762740.107809:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762740.107811:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762740.107813:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496165760:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762740.683298:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762741.707333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762742.731369:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762743.755312:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762744.779317:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762745.227273:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762745.227281:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762745.227285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762745.227293:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2724 00000100:00080000:1.0:1789762745.227303:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762745.227309:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762745.227343:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496165888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762745.227370:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762745.227373:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762745.227374:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762745.227377:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2724 00000100:00080000:1.0:1789762745.227380:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762745.227383:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762745.227391:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762745.227400:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496165888:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762745.227405:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496166016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762745.227419:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762745.227421:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762745.227423:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762745.227426:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2724 00000100:00080000:1.0:1789762745.227429:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762745.227431:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762745.227435:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762745.227443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496166144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762745.227450:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762745.227452:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762745.227454:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762745.227456:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2724 00000100:00080000:1.0:1789762745.227459:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762745.227461:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762745.227464:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762745.227470:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496166272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762745.227499:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496166016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762745.227523:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496166144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762745.227538:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496166272:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762745.803297:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762746.827361:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762747.851294:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762748.299519:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789762748.299578:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762748.299613:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762748.299658:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762748.299670:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762748.299687:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789762748.299747:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762745/real 1789762748] req@ffff8dd20504dc00 x1876699496166016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762748.299781:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496166016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762748.299790:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762748.299794:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762748.299797:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496166016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762748.299803:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762745/real 1789762748] req@ffff8dd20504c380 x1876699496166144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762748.299812:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496166144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762748.299818:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762748.299819:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762748.299821:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496166144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762748.299826:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762745/real 1789762748] req@ffff8dd20504ea00 x1876699496166272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762748.299833:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496166272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762748.299838:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762748.299841:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762748.299843:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496166272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762748.299847:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762745/real 1789762748] req@ffff8dd20504ca80 x1876699496165888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762748.299858:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496165888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762748.299864:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762748.299868:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762748.299871:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496165888:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789762748.875319:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762749.899314:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762750.347254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762750.347262:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762750.347266:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762750.347274:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2729 00000100:00080000:2.0:1789762750.347283:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762750.347289:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762750.347319:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496166400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762750.347349:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762750.347354:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762750.347356:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762750.347360:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2729 00000100:00080000:2.0:1789762750.347365:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762750.347369:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762750.347379:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762750.347403:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496166400:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762750.347413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496166528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762750.347424:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762750.347426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762750.347428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762750.347431:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2729 00000100:00080000:2.0:1789762750.347434:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762750.347437:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762750.347442:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762750.347451:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496166656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762750.347457:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762750.347460:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762750.347461:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762750.347463:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2729 00000100:00080000:2.0:1789762750.347465:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762750.347467:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762750.347470:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762750.347475:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762750.347480:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496166784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762750.347546:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762750/real 1789762750] req@ffff8dd205267800 x1876699496166400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762750.347566:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496166400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762750.347603:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762750.347608:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762750.347612:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496166400:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762750.347641:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496166528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762750.347670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762750.347684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762750/real 1789762750] req@ffff8dd205264a80 x1876699496166528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762750.347696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496166528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762750.347718:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762750.347722:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762750.347725:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496166528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762750.347732:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496166656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762750.347753:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762750.347764:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762750/real 1789762750] req@ffff8dd205267480 x1876699496166656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762750.347775:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496166656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762750.347784:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762750.347787:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762750.347790:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496166656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762750.347795:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496166784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762750.347813:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762750.347825:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762750/real 1789762750] req@ffff8dd205266680 x1876699496166784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762750.347840:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496166784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762750.347846:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762750.347848:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762750.347850:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496166784:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762750.923354:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762751.947308:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762752.971314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762753.995404:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762755.019314:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762755.467265:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762755.467274:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762755.467277:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762755.467286:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2735 00000100:00080000:0.0:1789762755.467294:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762755.467300:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762755.467330:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496166912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762755.467353:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762755.467356:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762755.467358:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762755.467361:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2735 00000100:00080000:0.0:1789762755.467363:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762755.467365:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762755.467373:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762755.467381:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496167040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762755.467394:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762755.467397:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762755.467398:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762755.467403:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2735 00000100:00100000:1.0:1789762755.467405:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496166912:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762755.467406:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762755.467410:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762755.467415:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762755.467429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496167168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762755.467441:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762755.467445:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762755.467447:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762755.467452:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2735 00000100:00080000:0.0:1789762755.467469:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762755.467476:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762755.467482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762755.467482:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762755.467498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496167296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762755.467521:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762755/real 1789762755] req@ffff8dd211834a80 x1876699496166912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762755.467536:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496166912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762755.467557:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762755.467560:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762755.467563:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496166912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762755.467584:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496167040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762755.467602:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762755.467612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762755/real 1789762755] req@ffff8dd211836680 x1876699496167040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762755.467620:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496167040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762755.467626:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762755.467628:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762755.467630:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496167040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762755.467634:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496167168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762755.467645:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762755.467652:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762755/real 1789762755] req@ffff8dd211834380 x1876699496167168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762755.467659:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496167168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762755.467665:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762755.467666:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762755.467668:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496167168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762755.467672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496167296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762755.467682:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762755.467688:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762755/real 1789762755] req@ffff8dd211835f80 x1876699496167296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762755.467695:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496167296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762755.467700:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762755.467702:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762755.467704:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496167296:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762756.043345:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762757.067327:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762758.091321:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762759.115297:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762760.139321:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762760.587300:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762760.587307:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762760.587311:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762760.587319:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2740 00000100:00080000:3.0:1789762760.587329:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762760.587335:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762760.587366:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496167424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762760.587388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762760.587391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762760.587392:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762760.587395:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2740 00000100:00080000:3.0:1789762760.587397:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762760.587399:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762760.587408:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762760.587426:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496167552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762760.587430:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496167424:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762760.587440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762760.587444:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762760.587445:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762760.587449:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2740 00000100:00080000:3.0:1789762760.587455:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762760.587457:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762760.587462:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762760.587471:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496167680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762760.587478:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762760.587480:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762760.587482:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762760.587484:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2740 00000100:00080000:3.0:1789762760.587486:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762760.587490:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762760.587491:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762760.587496:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762760.587520:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496167808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762760.587549:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762760/real 1789762760] req@ffff8dd203147480 x1876699496167424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762760.587563:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496167424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762760.587572:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762760.587575:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762760.587578:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496167424:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762760.587597:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496167552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762760.587613:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762760.587622:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762760/real 1789762760] req@ffff8dd203145880 x1876699496167552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762760.587632:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496167552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762760.587641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762760.587644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762760.587648:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496167552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762760.587654:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496167680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762760.587675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762760.587686:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762760/real 1789762760] req@ffff8dd203147100 x1876699496167680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762760.587695:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496167680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762760.587700:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762760.587703:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762760.587705:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496167680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762760.587709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496167808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762760.587722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762760.587729:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762760/real 1789762760] req@ffff8dd203144000 x1876699496167808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762760.587736:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496167808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762760.587741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762760.587743:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762760.587744:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496167808:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762761.163350:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762762.187306:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762763.211296:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762764.235326:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762765.259350:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762765.707252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762765.707263:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762765.707269:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762765.707279:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2745 00000100:00080000:3.0:1789762765.707288:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762765.707294:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762765.707338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496167936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762765.707363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762765.707366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762765.707368:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762765.707371:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2745 00000100:00080000:3.0:1789762765.707373:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762765.707376:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762765.707383:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762765.707393:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496168064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762765.707400:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762765.707403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762765.707404:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762765.707406:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2745 00000100:00080000:3.0:1789762765.707408:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762765.707411:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762765.707415:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762765.707417:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496167936:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762765.707424:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496168192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762765.707438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762765.707440:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762765.707442:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762765.707444:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2745 00000100:00080000:3.0:1789762765.707447:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762765.707450:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762765.707453:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762765.707460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496168320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762765.707491:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762765.707538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762765/real 1789762765] req@ffff8dd203145c00 x1876699496167936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762765.707553:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496167936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762765.707561:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762765.707563:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762765.707566:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496167936:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762765.707572:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496168064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762765.707599:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762765.707608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762765/real 1789762765] req@ffff8dd203145180 x1876699496168064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762765.707617:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496168064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762765.707622:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762765.707624:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762765.707627:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496168064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762765.707646:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496168192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762765.707658:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762765.707664:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762765/real 1789762765] req@ffff8dd203147800 x1876699496168192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762765.707672:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496168192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762765.707677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762765.707679:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762765.707681:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496168192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762765.707684:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496168320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762765.707694:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762765.707700:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762765/real 1789762765] req@ffff8dd203146300 x1876699496168320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762765.707712:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496168320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762765.707717:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762765.707719:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762765.707721:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496168320:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762766.283284:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762767.307318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762768.331306:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762769.355294:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762770.379319:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762770.827262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762770.827270:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762770.827273:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762770.827281:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2750 00000100:00080000:3.0:1789762770.827290:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762770.827296:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762770.827327:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496168448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762770.827349:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762770.827351:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762770.827353:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762770.827356:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2750 00000100:00080000:3.0:1789762770.827358:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762770.827360:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762770.827367:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762770.827375:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496168576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762770.827382:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762770.827384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762770.827385:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762770.827387:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2750 00000100:00080000:3.0:1789762770.827389:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762770.827391:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762770.827393:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762770.827399:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496168704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762770.827405:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762770.827407:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762770.827408:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762770.827411:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2750 00000100:00080000:3.0:1789762770.827413:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762770.827415:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762770.827418:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762770.827443:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496168448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762770.827444:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496168832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762770.827501:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762770.827533:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762770/real 1789762770] req@ffff8dd203144a80 x1876699496168448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762770.827546:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496168448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762770.827554:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762770.827557:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762770.827560:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496168448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762770.827565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496168576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762770.827578:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762770.827585:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762770/real 1789762770] req@ffff8dd203144700 x1876699496168576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762770.827597:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496168576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762770.827603:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762770.827605:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762770.827606:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496168576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762770.827610:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496168704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762770.827621:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762770.827626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762770/real 1789762770] req@ffff8dd203145f80 x1876699496168704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762770.827634:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496168704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762770.827639:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762770.827641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762770.827643:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496168704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762770.827663:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496168832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762770.827673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762770.827679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762770/real 1789762770] req@ffff8dd203144380 x1876699496168832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762770.827686:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496168832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762770.827696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762770.827698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762770.827700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496168832:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762771.403282:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762772.427328:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762773.451279:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762774.475340:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762775.499289:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762775.947341:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762775.947353:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762775.947359:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762775.947370:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2755 00000100:00080000:3.0:1789762775.947383:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762775.947417:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762775.947483:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496168960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762775.947517:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762775.947522:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762775.947524:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762775.947530:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2755 00000100:00080000:3.0:1789762775.947534:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762775.947538:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762775.947549:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762775.947563:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496169088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762775.947576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762775.947585:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762775.947587:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496168960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762775.947588:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762775.947601:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2755 00000100:00080000:3.0:1789762775.947609:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762775.947613:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762775.947620:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762775.947636:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496169216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762775.947656:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762775.947660:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762775.947662:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762775.947665:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2755 00000100:00080000:3.0:1789762775.947670:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762775.947674:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762775.947679:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762775.947687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762775.947696:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496169344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762775.947751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762775/real 1789762775] req@ffff8dd203145500 x1876699496168960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762775.947800:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496168960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762775.947819:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762775.947826:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762775.947833:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496168960:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762775.947868:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496169088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762775.947910:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762775.947933:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762775/real 1789762775] req@ffff8dd203146d80 x1876699496169088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762775.947955:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496169088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762775.947970:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762775.947977:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762775.947983:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496169088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789762775.947994:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496169216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789762775.948026:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762775.948044:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762775/real 1789762775] req@ffff8dd203147b80 x1876699496169216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762775.948063:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496169216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762775.948078:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762775.948083:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762775.948090:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496169216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789762775.948099:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496169344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789762775.948128:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762775.948146:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762775/real 1789762775] req@ffff8dd203146a00 x1876699496169344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762775.948166:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496169344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762775.948180:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762775.948240:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789762775.948249:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496169344:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762776.523397:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762777.547355:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762778.571334:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762779.595331:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762780.619305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762781.067290:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762781.067298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762781.067302:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762781.067312:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2760 00000100:00080000:1.0:1789762781.067321:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762781.067327:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762781.067372:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496169472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762781.067405:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762781.067408:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762781.067409:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762781.067413:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2760 00000100:00080000:1.0:1789762781.067416:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762781.067419:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762781.067428:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762781.067439:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496169600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762781.067447:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496169472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762781.067451:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762781.067455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762781.067456:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762781.067460:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2760 00000100:00080000:1.0:1789762781.067462:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762781.067465:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762781.067469:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762781.067478:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496169728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762781.067486:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762781.067488:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762781.067490:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762781.067492:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2760 00000100:00080000:1.0:1789762781.067496:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762781.067499:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762781.067501:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762781.067506:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762781.067510:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496169856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762781.067545:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762781/real 1789762781] req@ffff8dd20504c000 x1876699496169472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762781.067564:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496169472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762781.067572:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762781.067575:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762781.067578:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496169472:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762781.067596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496169600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762781.067611:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762781.067619:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762781/real 1789762781] req@ffff8dd20504d880 x1876699496169600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762781.067627:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496169600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762781.067633:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762781.067635:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762781.067637:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496169600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789762781.067641:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496169728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789762781.067652:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762781.067658:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762781/real 1789762781] req@ffff8dd20504ed80 x1876699496169728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762781.067665:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496169728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762781.067683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762781.067686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762781.067688:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496169728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789762781.067692:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496169856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789762781.067704:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762781.067711:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762781/real 1789762781] req@ffff8dd20504f100 x1876699496169856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762781.067718:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496169856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762781.067728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762781.067730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789762781.067732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496169856:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762781.643310:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762782.667332:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762783.691316:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762784.715335:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762785.739296:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762786.187258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762786.187269:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762786.187274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762786.187285:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2765 00000100:00080000:1.0:1789762786.187297:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762786.187305:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762786.187351:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496169984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762786.187376:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762786.187379:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762786.187381:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762786.187385:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2765 00000100:00080000:1.0:1789762786.187388:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762786.187390:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762786.187398:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762786.187408:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496170112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762786.187415:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762786.187418:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762786.187419:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762786.187422:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2765 00000100:00080000:1.0:1789762786.187423:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762786.187428:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762786.187431:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496169984:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789762786.187432:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762786.187445:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496170240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762786.187453:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762786.187455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762786.187456:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762786.187459:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2765 00000100:00080000:1.0:1789762786.187461:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762786.187464:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762786.187467:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762786.187488:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496170368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762786.187505:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762786.187540:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762786/real 1789762786] req@ffff8dd20504ce00 x1876699496169984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762786.187554:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496169984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762786.187562:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762786.187565:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762786.187568:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496169984:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762786.187573:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496170112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762786.187588:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762786.187596:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762786/real 1789762786] req@ffff8dd20504c700 x1876699496170112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762786.187604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496170112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762786.187610:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762786.187612:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762786.187614:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496170112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762786.187633:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496170240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762786.187644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762786.187650:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762786/real 1789762786] req@ffff8dd20504f800 x1876699496170240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762786.187657:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496170240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762786.187663:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762786.187664:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762786.187666:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496170240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762786.187670:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496170368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762786.187680:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762786.187686:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762786/real 1789762786] req@ffff8dd20504fb80 x1876699496170368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762786.187698:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496170368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762786.187703:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762786.187704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762786.187706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496170368:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762786.763337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762787.787339:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762788.811274:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762789.835338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762790.859346:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762791.307250:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762791.307256:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762791.307259:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762791.307267:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2770 00000100:00080000:3.0:1789762791.307287:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762791.307293:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762791.307321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496170496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762791.307340:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762791.307344:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762791.307345:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762791.307348:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2770 00000100:00080000:3.0:1789762791.307351:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762791.307353:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762791.307363:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762791.307378:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496170624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762791.307383:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496170496:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762791.307401:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762791.307405:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762791.307407:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762791.307412:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2770 00000100:00080000:3.0:1789762791.307416:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762791.307419:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762791.307425:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762791.307435:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496170752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762791.307443:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762791.307447:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762791.307449:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762791.307452:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762791.307456:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2770 00000100:00080000:3.0:1789762791.307460:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762791.307463:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762791.307466:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762791.307475:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496170880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762791.307494:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762791/real 1789762791] req@ffff8dd203144e00 x1876699496170496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762791.307510:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496170496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762791.307517:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762791.307520:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762791.307523:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496170496:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762791.307544:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496170624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762791.307560:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762791.307569:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762791/real 1789762791] req@ffff8dd203146a00 x1876699496170624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762791.307577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496170624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762791.307583:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762791.307585:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762791.307588:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496170624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762791.307591:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496170752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762791.307602:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762791.307607:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762791/real 1789762791] req@ffff8dd203147b80 x1876699496170752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762791.307615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496170752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762791.307620:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762791.307622:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762791.307624:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496170752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762791.307627:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496170880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762791.307638:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762791.307643:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762791/real 1789762791] req@ffff8dd203146d80 x1876699496170880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762791.307651:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496170880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762791.307656:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762791.307657:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762791.307660:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496170880:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762791.883301:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762792.907289:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762793.931307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762794.955308:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762795.979431:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762796.427339:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762796.427348:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762796.427352:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762796.427372:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2776 00000100:00080000:2.0:1789762796.427383:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762796.427390:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762796.427422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496171008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762796.427450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762796.427453:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762796.427454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762796.427457:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2776 00000100:00080000:2.0:1789762796.427460:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762796.427466:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762796.427476:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762796.427495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496171136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762796.427509:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496171008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762796.427519:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762796.427533:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762796.427536:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762796.427542:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2776 00000100:00080000:2.0:1789762796.427545:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762796.427549:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762796.427557:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762796.427569:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496171264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762796.427588:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762796.427590:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762796.427592:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762796.427594:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2776 00000100:00080000:2.0:1789762796.427596:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762796.427599:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762796.427602:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762796.427612:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496171392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762796.427641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762796.427733:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762796/real 1789762796] req@ffff8dd205265500 x1876699496171008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762796.427762:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496171008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762796.427795:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762796.427803:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762796.427810:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496171008:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762796.427863:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496171136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762796.427892:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762796.427905:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762796/real 1789762796] req@ffff8dd205266680 x1876699496171136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762796.427918:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496171136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762796.427927:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762796.427930:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762796.427934:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496171136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789762796.427957:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496171264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789762796.427975:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762796.427984:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762796/real 1789762796] req@ffff8dd205267480 x1876699496171264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762796.428000:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496171264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762796.428023:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762796.428028:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762796.428032:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496171264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789762796.428041:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496171392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762796.428069:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762796.428086:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762796/real 1789762796] req@ffff8dd205264a80 x1876699496171392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762796.428103:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496171392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762796.428110:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762796.428114:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762796.428117:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496171392:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762797.003392:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762798.027344:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762799.051304:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762800.075315:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762801.099378:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762801.547344:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762801.547358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762801.547364:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762801.547378:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2781 00000100:00080000:3.0:1789762801.547393:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762801.547403:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762801.547459:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496171520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762801.547502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762801.547507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762801.547511:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762801.547519:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2781 00000100:00080000:3.0:1789762801.547525:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762801.547530:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762801.547544:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762801.547546:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496171520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762801.547567:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496171648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762801.547583:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762801.547587:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762801.547589:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762801.547594:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2781 00000100:00080000:3.0:1789762801.547627:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762801.547637:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762801.547638:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762801.547652:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762801.547673:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496171776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762801.547692:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762801.547698:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762801.547701:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762801.547707:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762801/real 1789762801] req@ffff8dd203145500 x1876699496171520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762801.547715:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2781 00000100:00080000:3.0:1789762801.547731:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762801.547739:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762801.547746:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496171520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762801.547751:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762801.547771:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762801.547781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762801.547782:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496171904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762801.547796:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496171520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762801.547856:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496171648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762801.547887:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762801.547898:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762801/real 1789762801] req@ffff8dd203144380 x1876699496171648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762801.547907:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496171648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762801.547913:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762801.547916:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762801.547918:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496171648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762801.547923:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496171776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762801.547936:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762801.547942:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762801/real 1789762801] req@ffff8dd203145f80 x1876699496171776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762801.547950:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496171776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762801.547955:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762801.547957:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762801.547959:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496171776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762801.547973:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496171904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762801.547986:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762801.548005:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762801/real 1789762801] req@ffff8dd203144700 x1876699496171904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762801.548019:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496171904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762801.548028:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762801.548031:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762801.548035:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496171904:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762802.123327:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762803.147313:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762804.171326:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762805.195346:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762806.219304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762806.667286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762806.667297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762806.667302:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762806.667313:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2786 00000100:00080000:3.0:1789762806.667328:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762806.667338:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762806.667389:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496172032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762806.667416:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762806.667420:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762806.667421:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762806.667425:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2786 00000100:00080000:3.0:1789762806.667428:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762806.667431:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762806.667441:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762806.667453:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496172160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762806.667463:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762806.667468:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762806.667471:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762806.667474:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496172032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762806.667475:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2786 00000100:00080000:3.0:1789762806.667481:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762806.667489:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762806.667493:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762806.667503:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496172288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762806.667511:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762806.667514:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762806.667516:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762806.667519:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2786 00000100:00080000:3.0:1789762806.667522:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762806.667525:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762806.667529:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762806.667536:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496172416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762806.667551:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762806.667600:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762806/real 1789762806] req@ffff8dd203144a80 x1876699496172032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762806.667615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496172032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762806.667625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762806.667629:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762806.667634:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496172032:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762806.667654:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496172160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762806.667671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762806.667679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762806/real 1789762806] req@ffff8dd203146300 x1876699496172160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762806.667687:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496172160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762806.667693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762806.667695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762806.667697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496172160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762806.667701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496172288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762806.667713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762806.667719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762806/real 1789762806] req@ffff8dd203147800 x1876699496172288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762806.667728:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496172288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762806.667736:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762806.667740:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762806.667743:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496172288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762806.667749:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496172416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762806.667781:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762806.667793:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762806/real 1789762806] req@ffff8dd203145180 x1876699496172416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762806.667814:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496172416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762806.667823:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762806.667827:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762806.667830:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496172416:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762807.243290:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762808.267310:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762809.291342:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762810.315305:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762811.339296:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762811.787281:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762811.787289:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762811.787293:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762811.787302:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2791 00000100:00080000:1.0:1789762811.787310:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762811.787316:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762811.787349:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496172544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762811.787371:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762811.787374:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762811.787375:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762811.787379:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2791 00000100:00080000:1.0:1789762811.787382:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762811.787384:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762811.787392:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762811.787401:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496172672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762811.787409:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762811.787411:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762811.787412:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762811.787415:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2791 00000100:00080000:1.0:1789762811.787417:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762811.787419:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762811.787423:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762811.787429:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496172544:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762811.787444:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496172800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762811.787457:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762811.787461:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762811.787463:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762811.787472:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2791 00000100:00080000:1.0:1789762811.787477:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762811.787480:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762811.787485:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762811.787495:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496172928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762811.787542:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496172672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762811.787583:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496172800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762811.787609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496172928:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762812.363281:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762813.387343:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762814.411290:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789762814.859375:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789762814.859416:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789762814.859431:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762814.859458:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762814.859465:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789762814.859470:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1789762814.859497:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762811/real 1789762814] req@ffff8dd20504d180 x1876699496172672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762814.859517:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496172672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762814.859525:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762814.859529:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762814.859532:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496172672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762814.859539:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762811/real 1789762814] req@ffff8dd20504e300 x1876699496172800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762814.859547:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496172800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762814.859552:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762814.859554:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762814.859556:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496172800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762814.859559:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762811/real 1789762814] req@ffff8dd20504d500 x1876699496172928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762814.859566:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496172928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762814.859571:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762814.859572:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762814.859574:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496172928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762814.859578:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762811/real 1789762814] req@ffff8dd20504df80 x1876699496172544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762814.859585:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496172544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762814.859590:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762814.859592:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762814.859594:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496172544:192.168.203.109@tcp:250:kworker.0 00010000:00010000:1.0:1789762815.435299:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762816.459288:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762816.907231:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762816.907241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762816.907245:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762816.907256:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2796 00000100:00080000:3.0:1789762816.907266:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762816.907273:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762816.907307:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496173056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762816.907334:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762816.907338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762816.907340:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762816.907344:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2796 00000100:00080000:3.0:1789762816.907348:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762816.907353:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762816.907363:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762816.907375:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496173056:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762816.907379:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496173184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762816.907392:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762816.907396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762816.907398:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762816.907403:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2796 00000100:00080000:3.0:1789762816.907406:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762816.907410:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762816.907416:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762816.907427:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496173312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762816.907438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762816.907442:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789762816.907447:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762816.907447:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762816.907456:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2796 00000100:00080000:3.0:1789762816.907461:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762816.907465:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762816.907480:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762816.907495:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496173440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762816.907504:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762816/real 1789762816] req@ffff8dd203145c00 x1876699496173056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762816.907521:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496173056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762816.907529:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762816.907533:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762816.907537:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496173056:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762816.907562:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496173184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762816.907590:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762816.907604:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762816/real 1789762816] req@ffff8dd203144000 x1876699496173184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762816.907617:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496173184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762816.907627:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762816.907630:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762816.907634:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496173184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762816.907641:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496173312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762816.907665:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762816.907679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762816/real 1789762816] req@ffff8dd203147100 x1876699496173312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762816.907692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496173312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762816.907700:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762816.907704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762816.907707:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496173312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762816.907720:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496173440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762816.907743:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762816.907755:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762816/real 1789762816] req@ffff8dd203145880 x1876699496173440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762816.907766:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496173440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762816.907774:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762816.907777:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762816.907780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496173440:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762817.483305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762818.507282:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762819.531322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762820.555296:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762821.579295:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762822.027230:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762822.027239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762822.027243:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762822.027251:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2801 00000100:00080000:3.0:1789762822.027261:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762822.027267:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762822.027296:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496173568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762822.027317:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762822.027320:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762822.027321:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762822.027324:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2801 00000100:00080000:3.0:1789762822.027326:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762822.027328:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762822.027336:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762822.027349:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496173696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762822.027361:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762822.027367:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496173568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762822.027368:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762822.027372:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762822.027380:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2801 00000100:00080000:3.0:1789762822.027384:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762822.027389:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762822.027396:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762822.027409:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496173824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762822.027417:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762822.027420:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762822.027421:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762822.027425:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2801 00000100:00080000:3.0:1789762822.027427:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762822.027430:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762822.027433:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762822.027439:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762822.027445:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496173952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762822.027485:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762822/real 1789762822] req@ffff8dd203147480 x1876699496173568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762822.027513:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496173568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762822.027524:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762822.027528:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762822.027532:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496173568:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762822.027555:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496173696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762822.027580:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762822.027595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762822/real 1789762822] req@ffff8dd203145880 x1876699496173696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762822.027608:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496173696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762822.027616:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762822.027619:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762822.027623:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496173696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762822.027629:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496173824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762822.027659:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762822.027672:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762822/real 1789762822] req@ffff8dd203147100 x1876699496173824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762822.027684:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496173824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762822.027698:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762822.027703:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762822.027706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496173824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762822.027712:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496173952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762822.027731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762822.027742:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762822/real 1789762822] req@ffff8dd203144000 x1876699496173952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762822.027753:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496173952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762822.027761:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762822.027765:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762822.027768:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496173952:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762822.603287:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762823.627309:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762824.651270:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762825.675327:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762826.699286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762827.147244:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762827.147252:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762827.147256:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762827.147265:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2806 00000100:00080000:3.0:1789762827.147273:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762827.147280:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762827.147314:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496174080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762827.147342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762827.147346:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762827.147349:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762827.147353:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2806 00000100:00080000:3.0:1789762827.147357:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762827.147363:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762827.147377:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762827.147383:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496174080:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762827.147391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496174208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762827.147401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762827.147403:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762827.147405:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762827.147408:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2806 00000100:00080000:3.0:1789762827.147411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762827.147413:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762827.147417:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762827.147425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496174336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762827.147433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762827.147435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762827.147436:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762827.147440:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2806 00000800:00000100:2.0:1789762827.147444:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762827.147446:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762827.147451:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762827.147455:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762827.147463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496174464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762827.147500:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762827/real 1789762827] req@ffff8dd203145c00 x1876699496174080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.147523:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496174080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.147539:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.147542:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762827.147545:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496174080:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762827.147563:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496174208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762827.147582:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762827.147591:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762827/real 1789762827] req@ffff8dd203145180 x1876699496174208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.147600:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496174208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.147608:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.147612:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762827.147615:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496174208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762827.147622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496174336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762827.147642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762827.147655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762827/real 1789762827] req@ffff8dd203147800 x1876699496174336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.147668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496174336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.147677:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.147680:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762827.147684:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496174336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762827.147690:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496174464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762827.147724:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762827.147732:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762827/real 1789762827] req@ffff8dd203146300 x1876699496174464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762827.147740:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496174464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762827.147746:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762827.147748:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762827.147750:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496174464:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762827.723292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762828.747350:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762829.771305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762830.795367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762831.819349:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762832.267307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762832.267319:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762832.267323:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762832.267335:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2811 00000100:00080000:1.0:1789762832.267349:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762832.267356:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762832.267400:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496174592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762832.267429:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762832.267432:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762832.267433:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762832.267436:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2811 00000100:00080000:1.0:1789762832.267439:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762832.267441:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762832.267452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762832.267464:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496174720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762832.267472:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762832.267475:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762832.267477:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762832.267481:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2811 00000100:00080000:1.0:1789762832.267483:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762832.267486:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762832.267490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762832.267499:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496174848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762832.267510:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496174592:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762832.267514:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762832.267525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762832.267528:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762832.267532:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2811 00000100:00080000:1.0:1789762832.267535:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762832.267539:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762832.267544:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762832.267556:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496174976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762832.267595:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762832.267635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762832/real 1789762832] req@ffff8dd20504e680 x1876699496174592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762832.267655:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496174592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762832.267667:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762832.267671:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762832.267675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496174592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762832.267684:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496174720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762832.267717:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762832.267727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762832/real 1789762832] req@ffff8dd20504df80 x1876699496174720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762832.267735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496174720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762832.267741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762832.267744:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762832.267746:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496174720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762832.267767:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496174848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762832.267779:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762832.267785:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762832/real 1789762832] req@ffff8dd20504d500 x1876699496174848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762832.267793:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496174848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762832.267798:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762832.267800:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762832.267802:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496174848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762832.267806:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496174976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762832.267816:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762832.267822:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762832/real 1789762832] req@ffff8dd20504e300 x1876699496174976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762832.267834:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496174976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762832.267839:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762832.267841:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762832.267843:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496174976:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762832.843403:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762833.867287:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762834.891299:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762835.915305:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762836.939293:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762837.387260:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762837.387268:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762837.387273:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762837.387281:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2816 00000100:00080000:1.0:1789762837.387291:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762837.387298:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762837.387332:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496175104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762837.387351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762837.387353:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762837.387355:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762837.387358:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2816 00000100:00080000:1.0:1789762837.387361:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762837.387363:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762837.387371:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762837.387382:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496175104:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762837.387383:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496175232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762837.387397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762837.387399:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762837.387400:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762837.387402:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2816 00000100:00080000:1.0:1789762837.387405:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762837.387407:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762837.387410:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762837.387420:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496175360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762837.387431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762837.387435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762837.387437:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762837.387440:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2816 00000800:00000100:2.0:1789762837.387445:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762837.387446:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762837.387454:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762837.387460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762837.387475:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496175488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762837.387485:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762837/real 1789762837] req@ffff8dd20504d180 x1876699496175104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762837.387500:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496175104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762837.387508:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762837.387511:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762837.387514:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496175104:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762837.387532:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496175232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762837.387547:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762837.387555:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762837/real 1789762837] req@ffff8dd20504fb80 x1876699496175232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762837.387568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496175232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762837.387574:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762837.387576:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762837.387578:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496175232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762837.387582:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496175360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762837.387593:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762837.387599:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762837/real 1789762837] req@ffff8dd20504f800 x1876699496175360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762837.387606:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496175360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762837.387611:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762837.387613:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762837.387615:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496175360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762837.387618:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496175488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762837.387628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762837.387634:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762837/real 1789762837] req@ffff8dd20504c700 x1876699496175488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762837.387641:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496175488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762837.387651:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762837.387653:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762837.387655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496175488:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762837.963299:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762838.987330:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762840.011337:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762841.035300:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762842.059296:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762842.507261:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762842.507268:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762842.507272:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762842.507280:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2822 00000100:00080000:2.0:1789762842.507290:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762842.507297:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762842.507327:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496175616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762842.507349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762842.507351:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762842.507353:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762842.507356:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2822 00000100:00080000:2.0:1789762842.507358:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762842.507360:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762842.507367:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762842.507375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496175744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762842.507383:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762842.507386:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762842.507387:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762842.507393:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2822 00000100:00100000:3.0:1789762842.507393:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496175616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762842.507395:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762842.507401:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762842.507405:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762842.507424:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496175872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762842.507431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762842.507433:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762842.507435:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762842.507437:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2822 00000100:00080000:2.0:1789762842.507441:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762842.507444:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762842.507447:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762842.507454:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762842.507458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496176000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762842.507498:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762842/real 1789762842] req@ffff8dd205267800 x1876699496175616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762842.507514:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496175616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762842.507522:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762842.507525:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762842.507528:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496175616:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762842.507545:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496175744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762842.507561:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762842.507568:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762842/real 1789762842] req@ffff8dd205264380 x1876699496175744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762842.507576:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496175744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762842.507582:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762842.507584:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762842.507587:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496175744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789762842.507591:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496175872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789762842.507602:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762842.507608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762842/real 1789762842] req@ffff8dd205264e00 x1876699496175872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762842.507623:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496175872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762842.507628:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762842.507630:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762842.507633:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496175872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789762842.507636:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496176000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762842.507648:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762842.507654:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762842/real 1789762842] req@ffff8dd205267100 x1876699496176000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762842.507661:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496176000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762842.507666:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762842.507667:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762842.507669:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496176000:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762843.083294:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762844.107283:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762845.131312:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762846.155312:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762847.179295:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762847.627254:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762847.627265:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762847.627269:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762847.627281:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2827 00000100:00080000:2.0:1789762847.627293:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762847.627301:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762847.627340:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496176128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762847.627368:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762847.627372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762847.627374:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762847.627379:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2827 00000100:00080000:2.0:1789762847.627382:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762847.627387:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762847.627397:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762847.627409:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496176256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762847.627418:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496176128:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789762847.627425:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762847.627430:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762847.627432:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762847.627438:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2827 00000100:00080000:2.0:1789762847.627442:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762847.627447:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762847.627452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762847.627464:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496176384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762847.627476:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762847.627477:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762847.627483:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762847.627485:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762847.627491:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2827 00000100:00080000:2.0:1789762847.627495:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762847.627499:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762847.627504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762847.627516:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496176512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762847.627526:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762847/real 1789762847] req@ffff8dd205267b80 x1876699496176128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762847.627544:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496176128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762847.627553:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762847.627557:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762847.627560:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496176128:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762847.627578:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496176256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762847.627593:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762847.627601:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762847/real 1789762847] req@ffff8dd205264000 x1876699496176256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762847.627609:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496176256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762847.627615:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762847.627617:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762847.627619:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496176256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789762847.627622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496176384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789762847.627632:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762847.627638:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762847/real 1789762847] req@ffff8dd205265880 x1876699496176384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762847.627645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496176384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762847.627655:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762847.627657:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762847.627659:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496176384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789762847.627662:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496176512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762847.627672:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762847.627677:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762847/real 1789762847] req@ffff8dd205266a00 x1876699496176512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762847.627683:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496176512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762847.627688:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762847.627690:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762847.627691:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496176512:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762848.203303:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762849.227327:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762850.251297:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762851.275308:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762852.299358:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762852.747326:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762852.747336:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762852.747341:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762852.747352:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2832 00000100:00080000:1.0:1789762852.747364:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762852.747372:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762852.747417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496176640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762852.747449:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762852.747452:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762852.747453:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762852.747456:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2832 00000100:00080000:1.0:1789762852.747459:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762852.747461:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762852.747471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762852.747481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496176768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762852.747488:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762852.747491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762852.747493:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762852.747496:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2832 00000100:00080000:1.0:1789762852.747498:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762852.747501:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762852.747505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762852.747513:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496176640:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762852.747515:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496176896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762852.747530:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762852.747532:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762852.747533:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762852.747535:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2832 00000100:00080000:1.0:1789762852.747545:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762852.747548:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762852.747551:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762852.747560:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496177024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762852.747607:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762852.747667:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762852/real 1789762852] req@ffff8dd20504ce00 x1876699496176640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762852.747690:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496176640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762852.747702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762852.747706:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762852.747711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496176640:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762852.747721:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496176768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762852.747747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762852.747761:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762852/real 1789762852] req@ffff8dd20504f100 x1876699496176768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762852.747775:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496176768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762852.747784:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762852.747788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762852.747792:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496176768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789762852.747836:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496176896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789762852.747859:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762852.747871:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762852/real 1789762852] req@ffff8dd20504ed80 x1876699496176896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762852.747885:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496176896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762852.747894:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762852.747898:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762852.747901:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496176896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789762852.747908:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496177024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762852.747929:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762852.747939:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762852/real 1789762852] req@ffff8dd20504d880 x1876699496177024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762852.747952:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496177024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762852.747960:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762852.747963:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762852.747967:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496177024:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762853.323280:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762854.347392:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762855.371349:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762856.395389:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762857.419299:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762857.867265:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762857.867277:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762857.867281:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762857.867292:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2837 00000100:00080000:3.0:1789762857.867303:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762857.867311:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762857.867350:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496177152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762857.867380:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762857.867384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762857.867386:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762857.867390:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2837 00000100:00080000:3.0:1789762857.867395:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762857.867400:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762857.867409:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762857.867423:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496177152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762857.867425:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496177280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762857.867447:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762857.867451:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762857.867453:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762857.867456:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2837 00000100:00080000:3.0:1789762857.867460:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762857.867465:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762857.867471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762857.867484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496177408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762857.867493:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762857.867503:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762857.867507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762857.867509:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762857.867515:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2837 00000100:00080000:3.0:1789762857.867519:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762857.867524:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762857.867530:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762857.867531:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762857/real 1789762857] req@ffff8dd203144a80 x1876699496177152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762857.867545:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496177536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762857.867549:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496177152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762857.867578:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762857.867583:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762857.867588:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496177152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762857.867613:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496177280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762857.867635:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762857.867646:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762857/real 1789762857] req@ffff8dd203144700 x1876699496177280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762857.867654:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496177280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762857.867661:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762857.867663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762857.867665:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496177280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762857.867670:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496177408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762857.867682:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762857.867688:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762857/real 1789762857] req@ffff8dd203145f80 x1876699496177408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762857.867696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496177408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762857.867702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762857.867704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762857.867706:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496177408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762857.867709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496177536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762857.867719:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762857.867725:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762857/real 1789762857] req@ffff8dd203144380 x1876699496177536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762857.867732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496177536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762857.867738:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762857.867740:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762857.867742:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496177536:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762858.443311:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762859.467301:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762860.491422:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762861.515286:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762862.539420:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762862.987310:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762862.987320:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762862.987323:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762862.987331:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2842 00000100:00080000:1.0:1789762862.987342:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762862.987347:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762862.987387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496177664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762862.987417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762862.987421:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762862.987423:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762862.987428:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2842 00000100:00080000:1.0:1789762862.987431:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762862.987434:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762862.987447:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762862.987460:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496177792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762862.987467:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496177664:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762862.987471:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762862.987475:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762862.987476:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762862.987480:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2842 00000100:00080000:1.0:1789762862.987482:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762862.987485:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762862.987489:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762862.987502:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496177920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762862.987513:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762862.987517:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762862.987519:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762862.987522:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2842 00000100:00080000:1.0:1789762862.987526:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762862.987526:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762862.987535:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762862.987542:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762862.987557:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496178048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762862.987567:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762862/real 1789762862] req@ffff8dd20504c000 x1876699496177664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762862.987582:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496177664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762862.987591:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762862.987594:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762862.987597:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496177664:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762862.987616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496177792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762862.987631:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762862.987640:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762862/real 1789762862] req@ffff8dd20504ca80 x1876699496177792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762862.987649:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496177792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762862.987655:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762862.987657:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762862.987660:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496177792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762862.987664:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496177920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762862.987675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762862.987681:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762862/real 1789762862] req@ffff8dd20504ea00 x1876699496177920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762862.987688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496177920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762862.987693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762862.987695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762862.987697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496177920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762862.987700:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496178048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762862.987710:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762862.987716:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762862/real 1789762862] req@ffff8dd20504c380 x1876699496178048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762862.987723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496178048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762862.987728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762862.987730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762862.987732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496178048:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762863.563355:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762864.587330:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762865.611295:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762866.635305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762867.659318:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762868.107270:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762868.107280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762868.107284:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762868.107294:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2847 00000100:00080000:3.0:1789762868.107306:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762868.107314:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762868.107352:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496178176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762868.107379:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762868.107384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762868.107386:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762868.107391:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2847 00000100:00080000:3.0:1789762868.107396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762868.107400:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762868.107410:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762868.107414:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496178176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762868.107428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496178304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762868.107442:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762868.107446:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762868.107448:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762868.107451:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2847 00000100:00080000:3.0:1789762868.107464:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762868.107469:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762868.107475:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789762868.107481:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762868.107494:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496178432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762868.107505:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762868.107509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762868.107512:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762868.107516:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2847 00000100:00100000:2.0:1789762868.107521:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762868/real 1789762868] req@ffff8dd203145500 x1876699496178176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762868.107521:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762868.107531:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762868.107542:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762868.107552:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496178176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762868.107559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496178560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762868.107574:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762868.107580:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762868.107584:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496178176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762868.107606:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496178304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762868.107626:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762868.107635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762868/real 1789762868] req@ffff8dd203146d80 x1876699496178304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762868.107643:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496178304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762868.107649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762868.107652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762868.107654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496178304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762868.107658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496178432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762868.107669:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762868.107676:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762868/real 1789762868] req@ffff8dd203147b80 x1876699496178432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762868.107684:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496178432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762868.107689:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762868.107691:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762868.107693:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496178432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762868.107709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496178560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762868.107724:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762868.107731:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762868/real 1789762868] req@ffff8dd203146a00 x1876699496178560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762868.107742:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496178560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762868.107750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762868.107754:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762868.107757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496178560:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762868.683311:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762869.707288:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762870.731313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762871.755328:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762872.779291:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762873.227250:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762873.227259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762873.227263:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762873.227273:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2852 00000100:00080000:3.0:1789762873.227284:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762873.227290:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762873.227336:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496178688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762873.227359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762873.227363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762873.227364:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762873.227368:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2852 00000100:00080000:3.0:1789762873.227371:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762873.227373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762873.227381:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762873.227393:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496178816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762873.227406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762873.227409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762873.227411:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762873.227414:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2852 00000100:00080000:3.0:1789762873.227416:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762873.227419:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762873.227422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762873.227431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496178944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762873.227439:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762873.227443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762873.227444:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762873.227447:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2852 00000100:00080000:3.0:1789762873.227449:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762873.227455:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762873.227458:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496178688:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789762873.227460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762873.227471:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496179072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762873.227539:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762873.227599:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762873/real 1789762873] req@ffff8dd203144e00 x1876699496178688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762873.227624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496178688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762873.227633:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762873.227637:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762873.227640:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496178688:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762873.227647:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496178816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762873.227689:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762873.227719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762873/real 1789762873] req@ffff8dd203146a00 x1876699496178816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762873.227729:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496178816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762873.227736:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762873.227738:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762873.227741:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496178816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762873.227745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496178944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762873.227760:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762873.227768:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762873/real 1789762873] req@ffff8dd203147b80 x1876699496178944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762873.227776:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496178944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762873.227787:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762873.227789:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762873.227792:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496178944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762873.227811:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496179072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762873.227823:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762873.227830:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762873/real 1789762873] req@ffff8dd203146d80 x1876699496179072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762873.227839:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496179072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762873.227845:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762873.227847:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762873.227849:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496179072:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762873.803311:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762874.827306:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762875.851304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762876.875311:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762877.899323:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762878.347258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762878.347268:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762878.347273:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762878.347283:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2857 00000100:00080000:3.0:1789762878.347296:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762878.347304:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762878.347341:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496179200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762878.347363:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762878.347365:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762878.347367:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762878.347370:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2857 00000100:00080000:3.0:1789762878.347373:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762878.347376:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762878.347384:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762878.347398:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496179200:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762878.347398:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496179328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762878.347415:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762878.347417:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762878.347418:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762878.347420:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2857 00000100:00080000:3.0:1789762878.347423:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762878.347426:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762878.347429:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762878.347437:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496179456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762878.347444:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762878.347447:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762878.347449:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762878.347451:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2857 00000100:00080000:3.0:1789762878.347454:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762878.347456:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762878.347460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762878.347467:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496179584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762878.347511:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496179328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762878.347553:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496179456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762878.347575:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496179584:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762878.923292:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762879.947316:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762880.971297:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762881.419428:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789762881.419467:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762881.419481:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762881.419520:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762881.419528:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762881.419537:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1789762881.419574:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762878/real 1789762881] req@ffff8dd203144380 x1876699496179328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.419623:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496179328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.419636:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.419642:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762881.419648:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496179328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762881.419655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762878/real 1789762881] req@ffff8dd203145f80 x1876699496179456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.419668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496179456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.419676:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.419680:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762881.419683:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496179456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762881.419689:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762878/real 1789762881] req@ffff8dd203144700 x1876699496179584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.419701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496179584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.419709:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.419712:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762881.419714:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496179584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762881.419724:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762878/real 1789762881] req@ffff8dd203145500 x1876699496179200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762881.419734:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496179200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762881.419741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762881.419744:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762881.419747:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496179200:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789762881.995279:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762883.019333:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762883.467281:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762883.467289:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762883.467293:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762883.467301:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2863 00000100:00080000:1.0:1789762883.467316:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762883.467324:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762883.467359:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496179712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762883.467385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762883.467387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762883.467389:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762883.467392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2863 00000100:00080000:1.0:1789762883.467396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762883.467400:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762883.467409:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762883.467428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496179840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762883.467429:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496179712:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762883.467447:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762883.467451:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762883.467452:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762883.467457:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2863 00000100:00080000:1.0:1789762883.467461:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762883.467466:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762883.467472:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762883.467485:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496179968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762883.467497:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762883.467502:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762883.467503:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762883.467507:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762883.467514:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2863 00000100:00080000:1.0:1789762883.467521:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762883.467526:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762883.467532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762883.467541:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762883/real 1789762883] req@ffff8dd20504dc00 x1876699496179712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762883.467548:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496180096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762883.467574:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496179712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762883.467601:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762883.467606:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762883.467610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496179712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762883.467637:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496179840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762883.467664:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762883.467679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762883/real 1789762883] req@ffff8dd20504c380 x1876699496179840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762883.467693:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496179840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762883.467702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762883.467706:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762883.467710:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496179840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762883.467716:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496179968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762883.467738:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762883.467752:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762883/real 1789762883] req@ffff8dd20504ea00 x1876699496179968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762883.467765:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496179968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762883.467774:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762883.467777:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762883.467780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496179968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762883.467787:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496180096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762883.467814:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762883.467826:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762883/real 1789762883] req@ffff8dd20504ca80 x1876699496180096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762883.467839:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496180096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762883.467847:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762883.467850:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762883.467854:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496180096:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762884.043360:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762885.067354:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762886.091333:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762887.115311:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762888.139390:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762888.587421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762888.587447:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762888.587454:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762888.587470:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2868 00000100:00080000:2.0:1789762888.587485:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762888.587495:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762888.587542:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496180224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762888.587616:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762888.587624:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762888.587630:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762888.587640:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2868 00000100:00080000:2.0:1789762888.587649:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762888.587658:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762888.587674:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762888.587677:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496180224:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762888.587700:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496180352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762888.587718:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762888.587723:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762888.587726:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762888.587732:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2868 00000100:00080000:2.0:1789762888.587737:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762888.587744:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762888.587751:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762888.587766:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496180480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762888.587779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762888.587783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762888.587785:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762888.587795:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2868 00000800:00000100:3.0:1789762888.587798:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789762888.587820:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762888.587828:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762888.587835:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762888.587851:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496180608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762888.587895:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762888/real 1789762888] req@ffff8dd205264700 x1876699496180224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762888.587928:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496180224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762888.587939:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762888.587943:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762888.587947:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496180224:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762888.587999:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496180352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762888.588024:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762888.588035:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762888/real 1789762888] req@ffff8dd205266300 x1876699496180352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762888.588054:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496180352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762888.588062:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762888.588066:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762888.588070:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496180352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789762888.588077:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496180480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789762888.588101:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762888.588113:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762888/real 1789762888] req@ffff8dd205265180 x1876699496180480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762888.588127:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496180480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762888.588137:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762888.588141:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762888.588145:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496180480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789762888.588153:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496180608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762888.588200:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762888.588210:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762888/real 1789762888] req@ffff8dd205266d80 x1876699496180608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762888.588219:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496180608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762888.588225:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762888.588227:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762888.588230:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496180608:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762889.163366:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762890.187366:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762891.211292:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762892.235468:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762893.259391:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762893.707286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762893.707296:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762893.707300:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762893.707308:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2873 00000100:00080000:1.0:1789762893.707318:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762893.707325:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762893.707358:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496180736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762893.707381:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762893.707384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762893.707385:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762893.707389:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2873 00000100:00080000:1.0:1789762893.707391:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762893.707394:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762893.707402:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762893.707411:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496180864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762893.707424:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496180736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762893.707431:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762893.707435:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762893.707437:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762893.707440:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2873 00000100:00080000:1.0:1789762893.707443:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762893.707447:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762893.707450:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762893.707461:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496180992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762893.707469:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762893.707472:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762893.707473:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762893.707475:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2873 00000100:00080000:1.0:1789762893.707478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762893.707481:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762893.707484:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762893.707488:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789762893.707493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496181120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762893.707545:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762893/real 1789762893] req@ffff8dd20504c000 x1876699496180736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762893.707562:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496180736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762893.707574:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762893.707578:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762893.707582:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496180736:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762893.707609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496180864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762893.707632:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762893.707643:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762893/real 1789762893] req@ffff8dd20504d880 x1876699496180864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762893.707652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496180864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762893.707659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762893.707661:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762893.707664:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496180864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789762893.707668:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496180992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789762893.707681:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762893.707688:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762893/real 1789762893] req@ffff8dd20504ed80 x1876699496180992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762893.707696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496180992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762893.707701:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762893.707703:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762893.707705:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496180992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789762893.707709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496181120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762893.707719:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762893.707725:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762893/real 1789762893] req@ffff8dd20504f100 x1876699496181120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762893.707733:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496181120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762893.707739:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762893.707740:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762893.707742:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496181120:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762894.283263:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762895.307301:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762896.331446:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762897.355433:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762898.379282:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762898.827274:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762898.827283:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762898.827287:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762898.827296:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2878 00000100:00080000:3.0:1789762898.827307:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762898.827313:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762898.827348:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496181248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762898.827375:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762898.827379:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762898.827381:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762898.827385:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2878 00000100:00080000:3.0:1789762898.827389:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762898.827393:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762898.827402:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762898.827412:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496181248:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762898.827417:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496181376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762898.827430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762898.827433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762898.827435:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762898.827439:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2878 00000100:00080000:3.0:1789762898.827443:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762898.827447:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762898.827452:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762898.827463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496181504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762898.827483:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762898.827489:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762898.827494:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762898.827496:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762898.827502:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2878 00000100:00080000:3.0:1789762898.827505:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762898.827509:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762898.827514:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762898.827528:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496181632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762898.827531:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762898/real 1789762898] req@ffff8dd203144a80 x1876699496181248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762898.827568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496181248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762898.827577:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762898.827581:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762898.827584:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496181248:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762898.827602:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496181376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762898.827619:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762898.827627:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762898/real 1789762898] req@ffff8dd203146300 x1876699496181376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762898.827636:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496181376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762898.827641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762898.827643:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762898.827645:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496181376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762898.827649:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496181504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762898.827661:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762898.827666:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762898/real 1789762898] req@ffff8dd203147800 x1876699496181504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762898.827674:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496181504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762898.827683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762898.827686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762898.827688:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496181504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762898.827691:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496181632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762898.827701:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762898.827706:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762898/real 1789762898] req@ffff8dd203145180 x1876699496181632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762898.827713:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496181632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762898.827718:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762898.827720:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762898.827722:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496181632:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762899.403296:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762900.427286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762901.451501:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762902.475312:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762903.499292:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762903.947286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762903.947294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762903.947298:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762903.947306:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2883 00000100:00080000:0.0:1789762903.947315:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762903.947321:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762903.947356:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496181760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762903.947383:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762903.947387:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762903.947389:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762903.947394:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2883 00000100:00080000:0.0:1789762903.947399:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762903.947404:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762903.947413:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762903.947423:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496181760:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762903.947428:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496181888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762903.947453:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762903.947456:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762903.947457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762903.947460:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2883 00000100:00080000:0.0:1789762903.947464:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762903.947467:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762903.947472:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762903.947483:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496182016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762903.947496:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762903.947497:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789762903.947502:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762903.947504:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762903.947508:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2883 00000100:00080000:0.0:1789762903.947512:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762903.947515:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762903.947518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762903.947525:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496182144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762903.947536:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762903/real 1789762903] req@ffff8dd211834000 x1876699496181760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762903.947552:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496181760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762903.947560:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762903.947563:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762903.947567:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496181760:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762903.947585:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496181888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762903.947600:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762903.947608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762903/real 1789762903] req@ffff8dd211836a00 x1876699496181888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762903.947616:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496181888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762903.947621:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762903.947624:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762903.947626:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496181888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762903.947629:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496182016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762903.947639:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762903.947645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762903/real 1789762903] req@ffff8dd211835c00 x1876699496182016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762903.947652:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496182016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762903.947657:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762903.947659:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762903.947660:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496182016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762903.947664:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496182144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762903.947674:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762903.947680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762903/real 1789762903] req@ffff8dd211834700 x1876699496182144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762903.947687:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496182144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762903.947692:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762903.947694:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762903.947696:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496182144:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762904.523476:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762905.547335:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762906.571290:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762907.595286:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762908.619324:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762909.067243:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762909.067252:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762909.067255:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762909.067264:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2888 00000100:00080000:0.0:1789762909.067273:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762909.067279:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762909.067311:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496182272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762909.067333:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762909.067336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762909.067337:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762909.067341:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2888 00000100:00080000:0.0:1789762909.067343:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762909.067346:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762909.067353:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762909.067362:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496182400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762909.067370:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762909.067373:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762909.067374:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762909.067377:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2888 00000100:00080000:0.0:1789762909.067380:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762909.067384:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762909.067388:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762909.067392:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496182272:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789762909.067414:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496182528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762909.067424:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762909.067426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762909.067428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762909.067431:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2888 00000100:00080000:0.0:1789762909.067434:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762909.067437:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762909.067441:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762909.067450:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496182656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762909.067464:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762909.067502:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762909/real 1789762909] req@ffff8dd211837100 x1876699496182272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762909.067517:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496182272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762909.067526:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762909.067528:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762909.067532:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496182272:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762909.067539:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496182400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762909.067553:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762909.067561:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762909/real 1789762909] req@ffff8dd211836300 x1876699496182400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762909.067569:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496182400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762909.067574:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762909.067577:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762909.067579:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496182400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789762909.067597:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496182528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789762909.067608:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762909.067615:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762909/real 1789762909] req@ffff8dd211837800 x1876699496182528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762909.067622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496182528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762909.067627:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762909.067629:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762909.067631:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496182528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789762909.067634:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496182656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762909.067644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762909.067649:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762909/real 1789762909] req@ffff8dd211837b80 x1876699496182656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762909.067657:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496182656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762909.067662:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762909.067663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762909.067665:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496182656:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762909.643481:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762910.667307:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762911.691337:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762912.715337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762913.739301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762914.187258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762914.187266:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762914.187276:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762914.187285:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2893 00000100:00080000:3.0:1789762914.187294:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762914.187300:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762914.187333:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496182784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762914.187371:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762914.187374:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762914.187376:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762914.187379:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2893 00000100:00080000:3.0:1789762914.187381:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762914.187384:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762914.187404:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762914.187413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496182912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762914.187421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762914.187426:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496182784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762914.187426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762914.187429:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762914.187436:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2893 00000100:00080000:3.0:1789762914.187439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762914.187441:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762914.187444:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762914.187452:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496183040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762914.187470:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762914.187473:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762914.187474:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762914.187476:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2893 00000100:00080000:3.0:1789762914.187486:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762914.187489:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762914.187492:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762914.187500:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762914.187501:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496183168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762914.187538:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762914/real 1789762914] req@ffff8dd203145c00 x1876699496182784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762914.187552:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496182784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762914.187560:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762914.187563:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762914.187566:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496182784:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762914.187583:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496182912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762914.187600:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762914.187607:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762914/real 1789762914] req@ffff8dd203144000 x1876699496182912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762914.187615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496182912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762914.187621:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762914.187623:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762914.187625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496182912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762914.187629:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496183040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762914.187640:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762914.187646:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762914/real 1789762914] req@ffff8dd203147100 x1876699496183040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762914.187653:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496183040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762914.187658:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762914.187660:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762914.187661:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496183040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762914.187665:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496183168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762914.187675:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762914.187680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762914/real 1789762914] req@ffff8dd203145880 x1876699496183168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762914.187692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496183168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762914.187697:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762914.187699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762914.187701:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496183168:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762914.763287:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762915.787363:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762916.811403:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762917.835326:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762918.859424:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762919.307263:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762919.307272:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762919.307276:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762919.307285:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2898 00000100:00080000:1.0:1789762919.307295:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762919.307303:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762919.307355:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496183296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762919.307382:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762919.307386:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762919.307388:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762919.307392:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2898 00000100:00080000:1.0:1789762919.307396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762919.307401:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762919.307413:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762919.307417:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496183296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789762919.307429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496183424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762919.307441:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762919.307445:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762919.307447:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762919.307451:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2898 00000100:00080000:1.0:1789762919.307455:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762919.307460:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762919.307466:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762919.307478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496183552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762919.307484:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789762919.307495:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762919.307499:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762919.307502:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762919.307507:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2898 00000100:00080000:1.0:1789762919.307511:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762919.307516:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762919.307521:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762919.307524:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762919/real 1789762919] req@ffff8dd20504ce00 x1876699496183296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762919.307535:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496183680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762919.307543:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496183296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762919.307555:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762919.307559:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762919.307562:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496183296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762919.307581:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496183424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762919.307598:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762919.307606:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762919/real 1789762919] req@ffff8dd20504c700 x1876699496183424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762919.307619:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496183424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762919.307626:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762919.307630:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762919.307634:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496183424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762919.307639:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496183552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762919.307659:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762919.307671:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762919/real 1789762919] req@ffff8dd20504f800 x1876699496183552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762919.307683:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496183552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762919.307689:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762919.307691:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762919.307694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496183552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762919.307698:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496183680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762919.307712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762919.307719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762919/real 1789762919] req@ffff8dd20504fb80 x1876699496183680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762919.307727:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496183680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762919.307735:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762919.307737:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762919.307739:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496183680:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762919.883332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762920.907454:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762921.931297:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762922.955324:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762923.979319:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762924.427235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762924.427243:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762924.427247:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762924.427254:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2904 00000100:00080000:1.0:1789762924.427263:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762924.427269:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762924.427316:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496183808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762924.427343:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762924.427347:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762924.427349:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762924.427353:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2904 00000100:00080000:1.0:1789762924.427358:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762924.427362:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762924.427371:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762924.427384:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496183936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762924.427393:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762924.427398:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789762924.427398:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496183808:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762924.427400:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762924.427407:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2904 00000100:00080000:1.0:1789762924.427410:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762924.427413:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762924.427417:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762924.427425:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496184064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762924.427432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762924.427434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762924.427436:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762924.427438:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2904 00000100:00080000:1.0:1789762924.427440:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762924.427443:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762924.427445:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762924.427452:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496184192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762924.427463:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762924.427497:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762924/real 1789762924] req@ffff8dd20504d180 x1876699496183808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762924.427511:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496183808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762924.427518:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762924.427521:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762924.427524:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496183808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762924.427542:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496183936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762924.427565:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762924.427574:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762924/real 1789762924] req@ffff8dd20504e300 x1876699496183936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762924.427583:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496183936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762924.427593:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762924.427595:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762924.427598:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496183936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762924.427601:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496184064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762924.427613:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762924.427619:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762924/real 1789762924] req@ffff8dd20504d500 x1876699496184064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762924.427626:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496184064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762924.427632:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762924.427633:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762924.427635:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496184064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762924.427639:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496184192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762924.427649:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762924.427655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762924/real 1789762924] req@ffff8dd20504df80 x1876699496184192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762924.427662:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496184192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762924.427667:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762924.427669:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762924.427671:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496184192:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762925.003315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762926.027310:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762927.051323:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762928.075351:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762929.099418:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762929.547275:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762929.547285:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762929.547290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762929.547301:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2909 00000100:00080000:1.0:1789762929.547312:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762929.547319:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762929.547358:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496184320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762929.547384:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762929.547387:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762929.547388:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762929.547392:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2909 00000100:00080000:1.0:1789762929.547395:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762929.547398:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762929.547405:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762929.547418:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496184448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762929.547425:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762929.547428:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762929.547429:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762929.547431:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2909 00000100:00080000:1.0:1789762929.547434:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762929.547439:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496184320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762929.547440:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762929.547448:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762929.547458:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496184576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762929.547467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762929.547470:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762929.547471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762929.547474:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2909 00000100:00080000:1.0:1789762929.547478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762929.547480:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762929.547483:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762929.547490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496184704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762929.547499:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762929.547552:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762929/real 1789762929] req@ffff8dd20504e680 x1876699496184320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762929.547568:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496184320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762929.547577:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762929.547580:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762929.547583:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496184320:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762929.547587:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496184448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762929.547602:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762929.547612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762929/real 1789762929] req@ffff8dd20504df80 x1876699496184448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762929.547624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496184448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762929.547632:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762929.547636:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762929.547639:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496184448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762929.547665:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496184576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762929.547691:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762929.547700:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762929/real 1789762929] req@ffff8dd20504d500 x1876699496184576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762929.547707:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496184576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762929.547713:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762929.547715:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762929.547717:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496184576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762929.547721:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496184704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762929.547731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762929.547737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762929/real 1789762929] req@ffff8dd20504e300 x1876699496184704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762929.547744:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496184704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762929.547749:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762929.547751:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762929.547753:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496184704:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789762930.123307:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762931.147324:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762932.171312:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762933.195328:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762934.219333:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762934.667274:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762934.667283:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762934.667286:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762934.667295:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2914 00000100:00080000:3.0:1789762934.667304:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762934.667310:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762934.667342:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496184832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762934.667365:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762934.667368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762934.667369:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762934.667372:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2914 00000100:00080000:3.0:1789762934.667375:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762934.667377:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762934.667386:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762934.667396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496184960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762934.667404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762934.667408:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496184832:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762934.667409:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762934.667414:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762934.667421:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2914 00000100:00080000:3.0:1789762934.667425:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762934.667429:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762934.667435:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762934.667450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496185088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762934.667462:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762934.667465:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762934.667468:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762934.667469:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762934.667475:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2914 00000100:00080000:3.0:1789762934.667480:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762934.667483:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762934.667488:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762934.667498:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496185216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762934.667511:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762934/real 1789762934] req@ffff8dd203147480 x1876699496184832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762934.667539:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496184832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762934.667551:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762934.667556:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762934.667560:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496184832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762934.667582:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496184960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762934.667603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762934.667612:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762934/real 1789762934] req@ffff8dd203145880 x1876699496184960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762934.667621:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496184960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762934.667627:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762934.667630:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762934.667632:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496184960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762934.667636:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496185088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762934.667648:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762934.667654:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762934/real 1789762934] req@ffff8dd203147100 x1876699496185088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762934.667666:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496185088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762934.667671:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762934.667673:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762934.667675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496185088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762934.667678:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496185216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762934.667689:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762934.667695:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762934/real 1789762934] req@ffff8dd203144000 x1876699496185216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762934.667702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496185216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762934.667709:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762934.667712:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762934.667716:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496185216:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762935.243285:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762936.267320:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762937.291332:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762938.315309:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762939.339293:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762939.787259:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762939.787269:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762939.787275:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762939.787286:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2919 00000100:00080000:3.0:1789762939.787296:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762939.787302:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762939.787335:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496185344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762939.787357:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762939.787360:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762939.787361:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762939.787364:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2919 00000100:00080000:3.0:1789762939.787367:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762939.787370:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762939.787384:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762939.787388:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496185344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762939.787396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496185472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762939.787405:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762939.787407:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762939.787409:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762939.787411:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2919 00000100:00080000:3.0:1789762939.787415:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762939.787417:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762939.787421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762939.787428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496185600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762939.787440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762939.787445:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762939.787446:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762939.787450:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762939.787453:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2919 00000100:00080000:3.0:1789762939.787457:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762939.787459:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762939.787463:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762939.787475:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496185728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762939.787484:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762939/real 1789762939] req@ffff8dd203145c00 x1876699496185344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762939.787505:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496185344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762939.787513:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762939.787516:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762939.787519:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496185344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762939.787536:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496185472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762939.787550:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762939.787559:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762939/real 1789762939] req@ffff8dd203145180 x1876699496185472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762939.787567:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496185472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762939.787572:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762939.787574:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762939.787577:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496185472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762939.787580:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496185600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762939.787591:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762939.787597:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762939/real 1789762939] req@ffff8dd203147800 x1876699496185600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762939.787604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496185600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762939.787613:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762939.787616:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762939.787617:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496185600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762939.787621:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496185728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762939.787630:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762939.787636:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762939/real 1789762939] req@ffff8dd203146300 x1876699496185728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762939.787642:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496185728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762939.787648:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762939.787651:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762939.787655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496185728:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762940.363297:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762941.387356:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762942.411348:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762943.435302:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762944.459305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762944.907271:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762944.907279:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762944.907282:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762944.907291:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2924 00000100:00080000:3.0:1789762944.907300:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762944.907306:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762944.907343:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496185856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762944.907365:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762944.907368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762944.907369:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762944.907372:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2924 00000100:00080000:3.0:1789762944.907375:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762944.907377:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762944.907386:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762944.907395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496185984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762944.907402:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762944.907405:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762944.907406:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762944.907409:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2924 00000100:00080000:3.0:1789762944.907411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762944.907415:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496185856:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762944.907416:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762944.907422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762944.907434:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496186112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762944.907445:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762944.907449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762944.907451:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762944.907455:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2924 00000100:00080000:3.0:1789762944.907459:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762944.907463:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762944.907469:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762944.907482:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496186240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762944.907527:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496185984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762944.907560:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496186112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762944.907576:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496186240:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762945.483306:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762946.507328:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762947.531330:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762947.979409:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789762947.979451:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762947.979468:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762947.979507:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762947.979516:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789762947.979524:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1789762947.979552:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762944/real 1789762947] req@ffff8dd203144700 x1876699496186112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.979573:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496186112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.979582:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.979586:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762947.979589:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496186112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762947.979596:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762944/real 1789762947] req@ffff8dd203145f80 x1876699496186240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.979609:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496186240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.979617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.979621:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762947.979624:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496186240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762947.979630:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762944/real 1789762947] req@ffff8dd203144a80 x1876699496185856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.979641:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496185856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.979654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.979657:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762947.979661:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496185856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762947.979667:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762944/real 1789762947] req@ffff8dd203145500 x1876699496185984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762947.979680:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496185984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762947.979688:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762947.979691:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762947.979695:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496185984:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762948.555369:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762949.579415:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762950.027278:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762950.027293:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762950.027300:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762950.027317:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2929 00000100:00080000:3.0:1789762950.027352:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762950.027365:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762950.027416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496186368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762950.027486:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762950.027493:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762950.027497:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762950.027505:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2929 00000100:00080000:3.0:1789762950.027513:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762950.027522:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762950.027538:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762950.027569:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496186496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762950.027582:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496186368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762950.027593:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762950.027629:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762950.027632:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762950.027642:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2929 00000100:00080000:3.0:1789762950.027651:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762950.027659:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762950.027671:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762950.027702:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496186624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762950.027723:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789762950.027748:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762950.027756:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762950.027760:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762950.027768:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2929 00000100:00080000:3.0:1789762950.027776:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762950.027783:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762950.027794:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762950.027797:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762950/real 1789762950] req@ffff8dd203144380 x1876699496186368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762950.027827:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496186752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762950.027839:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496186368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762950.027870:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762950.027879:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762950.027889:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496186368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762950.027927:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496186496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762950.027973:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762950.028000:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762950/real 1789762950] req@ffff8dd203146d80 x1876699496186496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762950.028043:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496186496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762950.028060:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762950.028067:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762950.028073:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496186496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762950.028086:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496186624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762950.028160:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762950.028222:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762950/real 1789762950] req@ffff8dd203147b80 x1876699496186624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762950.028247:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496186624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762950.028264:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762950.028270:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762950.028277:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496186624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762950.028289:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496186752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762950.028327:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762950.028378:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762950/real 1789762950] req@ffff8dd203146a00 x1876699496186752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762950.028409:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496186752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762950.028439:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762950.028445:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762950.028451:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496186752:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762950.603321:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762951.627327:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762952.651362:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762953.675294:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762954.699352:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762955.147213:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762955.147221:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762955.147224:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762955.147244:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2934 00000100:00080000:3.0:1789762955.147254:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762955.147259:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762955.147296:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496186880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762955.147332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762955.147336:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762955.147338:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762955.147343:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2934 00000100:00080000:3.0:1789762955.147347:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762955.147352:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762955.147363:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762955.147377:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496186880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762955.147380:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496187008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762955.147392:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762955.147394:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762955.147406:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762955.147409:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2934 00000100:00080000:3.0:1789762955.147412:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762955.147415:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762955.147419:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762955.147428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496187136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762955.147436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762955.147439:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762955.147440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762955.147443:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2934 00000100:00080000:3.0:1789762955.147445:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762955.147448:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762955.147450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762955.147459:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496187264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762955.147462:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762955.147518:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762955/real 1789762955] req@ffff8dd203144e00 x1876699496186880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762955.147549:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496186880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762955.147560:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762955.147564:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762955.147568:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496186880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762955.147592:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496187008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762955.147618:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762955.147633:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762955/real 1789762955] req@ffff8dd203146a00 x1876699496187008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762955.147648:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496187008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762955.147664:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762955.147668:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762955.147672:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496187008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762955.147678:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496187136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762955.147724:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762955.147737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762955/real 1789762955] req@ffff8dd203147b80 x1876699496187136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762955.147749:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496187136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762955.147758:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762955.147762:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762955.147765:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496187136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762955.147771:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496187264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762955.147793:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762955.147805:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762955/real 1789762955] req@ffff8dd203146d80 x1876699496187264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762955.147818:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496187264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762955.147826:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762955.147829:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762955.147832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496187264:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762955.723307:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762956.747400:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762957.771303:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762958.795292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762959.819297:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762960.267264:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762960.267272:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762960.267276:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762960.267284:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2939 00000100:00080000:1.0:1789762960.267298:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762960.267305:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762960.267337:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496187392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762960.267357:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762960.267361:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762960.267363:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762960.267366:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2939 00000100:00080000:1.0:1789762960.267368:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762960.267371:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762960.267379:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762960.267388:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496187520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762960.267394:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496187392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762960.267399:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762960.267402:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762960.267404:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762960.267407:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2939 00000100:00080000:1.0:1789762960.267409:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762960.267412:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762960.267415:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762960.267424:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496187648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762960.267431:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762960.267434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762960.267435:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762960.267437:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2939 00000100:00080000:1.0:1789762960.267446:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762960.267450:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1789762960.267454:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789762960.267456:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762960.267466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496187776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762960.267496:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762960/real 1789762960] req@ffff8dd20504d180 x1876699496187392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762960.267509:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496187392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762960.267516:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762960.267519:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762960.267522:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496187392:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762960.267539:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496187520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762960.267554:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762960.267563:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762960/real 1789762960] req@ffff8dd20504fb80 x1876699496187520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762960.267571:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496187520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762960.267576:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762960.267578:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762960.267581:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496187520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762960.267585:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496187648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762960.267596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762960.267602:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762960/real 1789762960] req@ffff8dd20504f800 x1876699496187648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762960.267609:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496187648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762960.267615:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762960.267616:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762960.267619:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496187648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762960.267624:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496187776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762960.267642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762960.267653:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762960/real 1789762960] req@ffff8dd20504c700 x1876699496187776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762960.267666:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496187776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762960.267675:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762960.267677:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762960.267680:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496187776:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762960.843276:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762961.867312:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762962.891284:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762963.915297:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762964.939282:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762965.387361:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762965.387371:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762965.387376:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762965.387387:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2944 00000100:00080000:0.0:1789762965.387397:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762965.387404:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789762965.387443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496187904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762965.387469:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762965.387472:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762965.387474:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762965.387477:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2944 00000100:00080000:0.0:1789762965.387481:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762965.387483:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762965.387492:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762965.387505:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496188032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762965.387515:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762965.387520:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762965.387524:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762965.387524:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496187904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789762965.387534:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2944 00000100:00080000:0.0:1789762965.387537:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762965.387541:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762965.387545:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762965.387558:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496188160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762965.387567:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762965.387570:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762965.387572:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762965.387575:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2944 00000100:00080000:0.0:1789762965.387578:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762965.387581:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789762965.387586:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762965.387596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789762965.387598:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496188288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762965.387664:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762965/real 1789762965] req@ffff8dd211837b80 x1876699496187904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762965.387689:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496187904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762965.387702:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762965.387708:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762965.387713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496187904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762965.387743:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496188032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762965.387772:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762965.387787:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762965/real 1789762965] req@ffff8dd211837800 x1876699496188032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762965.387803:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496188032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762965.387813:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762965.387817:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762965.387821:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496188032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762965.387828:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496188160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762965.387847:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762965.387858:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762965/real 1789762965] req@ffff8dd211836300 x1876699496188160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762965.387881:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496188160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762965.387891:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762965.387894:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762965.387898:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496188160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762965.387904:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496188288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762965.387922:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762965.387933:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762965/real 1789762965] req@ffff8dd211837100 x1876699496188288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762965.387947:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496188288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762965.387956:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762965.387959:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762965.387963:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496188288:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762965.963339:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762966.987401:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762968.011421:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762969.035318:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762970.059392:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762970.507266:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762970.507277:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762970.507281:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762970.507293:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2950 00000100:00080000:2.0:1789762970.507305:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762970.507314:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789762970.507354:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496188416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762970.507379:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762970.507382:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762970.507383:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762970.507387:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2950 00000100:00080000:2.0:1789762970.507390:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762970.507395:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762970.507404:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762970.507411:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496188416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762970.507419:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496188544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762970.507435:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762970.507438:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762970.507439:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762970.507442:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2950 00000100:00080000:2.0:1789762970.507444:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762970.507447:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762970.507450:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762970.507458:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496188672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762970.507465:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762970.507468:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762970.507470:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762970.507472:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2950 00000100:00080000:2.0:1789762970.507475:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762970.507477:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789762970.507481:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762970.507482:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762970.507492:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496188800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762970.507524:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762970/real 1789762970] req@ffff8dd205265f80 x1876699496188416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762970.507545:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496188416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762970.507556:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762970.507561:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762970.507565:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496188416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789762970.507587:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496188544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762970.507612:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762970.507627:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762970/real 1789762970] req@ffff8dd205266d80 x1876699496188544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762970.507641:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496188544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762970.507647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762970.507649:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762970.507651:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496188544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789762970.507656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496188672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789762970.507671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762970.507679:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762970/real 1789762970] req@ffff8dd205265180 x1876699496188672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762970.507691:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496188672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762970.507696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762970.507698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762970.507700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496188672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789762970.507703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496188800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789762970.507713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762970.507719:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762970/real 1789762970] req@ffff8dd205266300 x1876699496188800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762970.507726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496188800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762970.507731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762970.507733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789762970.507734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496188800:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762971.083293:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762972.107329:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762973.131431:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762974.155393:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762975.179396:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762975.627417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762975.627429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762975.627435:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762975.627449:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2955 00000100:00080000:3.0:1789762975.627462:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762975.627470:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762975.627516:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496188928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762975.627556:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762975.627561:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762975.627563:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762975.627568:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2955 00000100:00080000:3.0:1789762975.627572:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762975.627576:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762975.627587:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762975.627604:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496189056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762975.627614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496188928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762975.627623:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762975.627630:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762975.627632:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762975.627638:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2955 00000100:00080000:3.0:1789762975.627641:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762975.627645:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762975.627651:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762975.627666:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496189184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762975.627687:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762975.627691:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762975.627694:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762975.627698:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2955 00000100:00080000:3.0:1789762975.627704:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762975.627708:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762975.627713:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762975.627720:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762975.627732:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496189312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762975.627791:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762975/real 1789762975] req@ffff8dd203144380 x1876699496188928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762975.627815:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496188928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762975.627828:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762975.627833:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762975.627839:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496188928:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762975.627867:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496189056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762975.627898:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762975.627942:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762975/real 1789762975] req@ffff8dd203145500 x1876699496189056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762975.627959:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496189056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762975.627970:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762975.627974:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762975.627978:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496189056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762975.627985:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496189184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762975.628011:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762975.628024:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762975/real 1789762975] req@ffff8dd203144a80 x1876699496189184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762975.628038:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496189184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762975.628047:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762975.628051:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762975.628055:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496189184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762975.628061:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496189312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762975.628079:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762975.628089:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762975/real 1789762975] req@ffff8dd203145f80 x1876699496189312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762975.628102:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496189312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762975.628119:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762975.628123:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762975.628127:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496189312:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762976.203365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762977.227402:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762978.251419:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762979.275306:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762980.299278:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762980.747291:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762980.747302:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762980.747307:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762980.747318:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2960 00000100:00080000:3.0:1789762980.747330:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762980.747337:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762980.747377:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496189440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762980.747408:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762980.747412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762980.747415:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762980.747419:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2960 00000100:00080000:3.0:1789762980.747423:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762980.747427:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762980.747436:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762980.747457:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496189568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762980.747467:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762980.747472:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762980.747474:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762980.747478:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2960 00000100:00080000:3.0:1789762980.747481:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762980.747485:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762980.747490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762980.747499:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496189696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762980.747511:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762980.747515:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789762980.747518:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496189440:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762980.747518:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762980.747525:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2960 00000100:00080000:3.0:1789762980.747550:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762980.747555:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762980.747559:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762980.747568:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496189824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762980.747639:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762980.747686:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762980/real 1789762980] req@ffff8dd203144700 x1876699496189440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762980.747706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496189440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762980.747717:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762980.747721:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762980.747725:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496189440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762980.747733:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496189568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762980.747755:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762980.747766:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762980/real 1789762980] req@ffff8dd203146300 x1876699496189568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762980.747778:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496189568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762980.747793:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762980.747796:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762980.747817:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496189568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762980.747838:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496189696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762980.747854:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762980.747862:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762980/real 1789762980] req@ffff8dd203147800 x1876699496189696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762980.747893:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496189696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762980.747900:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762980.747903:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762980.747906:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496189696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762980.747911:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496189824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762980.747928:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762980.747937:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762980/real 1789762980] req@ffff8dd203145180 x1876699496189824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762980.747947:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496189824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762980.747953:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762980.747956:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762980.747959:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496189824:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762981.323399:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762982.347426:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762983.371410:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762984.395376:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762985.419294:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762985.867289:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762985.867297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762985.867301:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762985.867315:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2965 00000100:00080000:3.0:1789762985.867325:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762985.867331:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789762985.867364:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496189952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762985.867386:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762985.867390:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762985.867392:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762985.867397:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2965 00000100:00080000:3.0:1789762985.867400:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762985.867404:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762985.867425:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762985.867438:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496190080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762985.867449:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762985.867450:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496189952:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789762985.867452:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762985.867454:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762985.867457:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2965 00000100:00080000:3.0:1789762985.867461:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762985.867464:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762985.867469:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762985.867477:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496190208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762985.867486:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762985.867488:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762985.867490:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762985.867493:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2965 00000100:00080000:3.0:1789762985.867496:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762985.867510:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789762985.867514:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762985.867518:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789762985.867524:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496190336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762985.867559:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762985/real 1789762985] req@ffff8dd203145c00 x1876699496189952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762985.867577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496189952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762985.867598:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762985.867600:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762985.867603:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496189952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762985.867623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496190080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762985.867641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762985.867650:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762985/real 1789762985] req@ffff8dd203144000 x1876699496190080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762985.867667:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496190080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762985.867672:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762985.867673:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762985.867675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496190080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762985.867680:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496190208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762985.867690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762985.867696:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762985/real 1789762985] req@ffff8dd203147100 x1876699496190208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762985.867703:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496190208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762985.867709:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762985.867710:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762985.867712:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496190208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762985.867727:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496190336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762985.867738:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762985.867743:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762985/real 1789762985] req@ffff8dd203145880 x1876699496190336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762985.867750:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496190336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762985.867755:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762985.867757:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762985.867759:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496190336:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789762986.443301:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762987.467297:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762988.491322:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762989.515322:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762990.539318:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762990.987255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762990.987263:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762990.987267:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762990.987275:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2970 00000100:00080000:1.0:1789762990.987284:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762990.987289:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762990.987318:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496190464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762990.987338:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762990.987341:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762990.987343:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762990.987346:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2970 00000100:00080000:1.0:1789762990.987348:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762990.987351:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762990.987358:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762990.987368:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496190592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762990.987375:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496190464:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762990.987378:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762990.987383:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762990.987384:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762990.987387:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2970 00000100:00080000:1.0:1789762990.987390:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762990.987393:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762990.987397:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762990.987406:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496190720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762990.987414:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762990.987416:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762990.987417:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762990.987419:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2970 00000100:00080000:1.0:1789762990.987422:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762990.987424:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762990.987427:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762990.987435:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496190848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762990.987439:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762990.987485:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762990/real 1789762990] req@ffff8dd20504ce00 x1876699496190464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762990.987503:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496190464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762990.987511:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762990.987514:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762990.987517:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496190464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762990.987537:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496190592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762990.987560:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762990.987575:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762990/real 1789762990] req@ffff8dd20504f100 x1876699496190592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762990.987589:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496190592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762990.987598:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762990.987602:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762990.987604:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496190592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762990.987609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496190720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762990.987627:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762990.987635:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762990/real 1789762990] req@ffff8dd20504ed80 x1876699496190720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762990.987648:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496190720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762990.987654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762990.987656:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762990.987658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496190720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762990.987661:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496190848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762990.987672:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762990.987678:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762990/real 1789762990] req@ffff8dd20504d880 x1876699496190848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762990.987686:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496190848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762990.987691:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762990.987693:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762990.987694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496190848:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762991.563384:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762992.587429:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762993.611371:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762994.635416:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762995.659358:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762996.107257:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762996.107268:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762996.107272:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762996.107283:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2975 00000100:00080000:1.0:1789762996.107296:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762996.107303:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789762996.107349:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496190976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762996.107385:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762996.107391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762996.107392:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762996.107395:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2975 00000100:00080000:1.0:1789762996.107398:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762996.107401:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762996.107411:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762996.107421:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496191104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762996.107428:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762996.107431:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762996.107433:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762996.107437:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2975 00000100:00080000:1.0:1789762996.107441:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762996.107444:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762996.107448:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762996.107455:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496191232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762996.107465:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762996.107470:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762996.107475:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762996.107476:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496190976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789762996.107480:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2975 00000100:00080000:1.0:1789762996.107493:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762996.107498:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789762996.107502:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762996.107515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496191360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762996.107556:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762996.107595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762996/real 1789762996] req@ffff8dd20504c000 x1876699496190976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762996.107609:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496190976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762996.107619:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762996.107622:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762996.107625:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496190976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789762996.107630:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496191104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762996.107645:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762996.107653:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762996/real 1789762996] req@ffff8dd20504ca80 x1876699496191104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762996.107660:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496191104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762996.107666:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762996.107668:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762996.107670:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496191104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789762996.107689:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496191232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789762996.107700:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762996.107707:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762996/real 1789762996] req@ffff8dd20504ea00 x1876699496191232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762996.107714:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496191232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762996.107719:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762996.107721:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762996.107723:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496191232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789762996.107727:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496191360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789762996.107737:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789762996.107743:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762996/real 1789762996] req@ffff8dd20504c380 x1876699496191360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762996.107750:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496191360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762996.107771:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762996.107773:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789762996.107775:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496191360:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789762996.683303:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762997.707287:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762998.731346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762999.755326:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763000.779290:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763001.227246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763001.227255:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763001.227259:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763001.227269:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2980 00000100:00080000:1.0:1789763001.227280:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763001.227287:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763001.227319:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496191488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763001.227347:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763001.227355:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763001.227357:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763001.227361:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2980 00000100:00080000:1.0:1789763001.227364:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763001.227368:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763001.227377:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763001.227391:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496191616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763001.227399:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496191488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763001.227405:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763001.227410:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763001.227412:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763001.227416:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2980 00000100:00080000:1.0:1789763001.227421:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763001.227425:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763001.227431:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763001.227443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496191744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763001.227454:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763001.227458:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763001.227461:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789763001.227466:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763001.227470:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2980 00000100:00080000:1.0:1789763001.227476:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763001.227480:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763001.227486:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763001.227497:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496191872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763001.227506:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763001/real 1789763001] req@ffff8dd20504dc00 x1876699496191488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763001.227521:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496191488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763001.227530:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763001.227533:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763001.227536:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496191488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763001.227554:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496191616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763001.227571:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763001.227579:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763001/real 1789763001] req@ffff8dd20504c380 x1876699496191616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763001.227587:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496191616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763001.227593:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763001.227595:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763001.227597:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496191616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763001.227601:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496191744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763001.227611:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763001.227617:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763001/real 1789763001] req@ffff8dd20504ea00 x1876699496191744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763001.227624:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496191744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763001.227629:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763001.227631:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763001.227632:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496191744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763001.227636:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496191872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763001.227645:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763001.227651:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763001/real 1789763001] req@ffff8dd20504ca80 x1876699496191872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763001.227663:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496191872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763001.227669:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763001.227670:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763001.227672:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496191872:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763001.803256:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763002.827320:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763003.851381:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763004.875365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763005.899313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763006.347265:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763006.347279:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763006.347285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763006.347299:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2985 00000100:00080000:1.0:1789763006.347313:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763006.347323:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763006.347374:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496192000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763006.347408:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763006.347415:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763006.347417:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763006.347422:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2985 00000100:00080000:1.0:1789763006.347426:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763006.347430:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763006.347442:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763006.347457:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496192000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763006.347460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496192128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763006.347475:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763006.347479:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763006.347482:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763006.347486:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2985 00000100:00080000:1.0:1789763006.347490:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763006.347495:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763006.347502:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763006.347515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496192256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763006.347527:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763006.347531:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789763006.347535:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763006.347536:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763006.347543:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2985 00000100:00080000:1.0:1789763006.347549:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763006.347554:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763006.347562:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763006.347576:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496192384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763006.347586:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763006/real 1789763006] req@ffff8dd20504c000 x1876699496192000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763006.347610:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496192000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763006.347622:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763006.347627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763006.347632:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496192000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763006.347661:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496192128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763006.347695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763006.347707:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763006/real 1789763006] req@ffff8dd20504d880 x1876699496192128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763006.347721:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496192128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763006.347727:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763006.347730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763006.347732:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496192128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763006.347737:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496192256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763006.347751:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763006.347758:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763006/real 1789763006] req@ffff8dd20504ed80 x1876699496192256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763006.347766:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496192256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763006.347771:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763006.347773:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763006.347775:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496192256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763006.347779:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496192384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763006.347789:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763006.347794:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763006/real 1789763006] req@ffff8dd20504f100 x1876699496192384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763006.347801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496192384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763006.347814:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763006.347817:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763006.347821:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496192384:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763006.923312:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763007.947323:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763008.971289:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763009.995322:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763011.019346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763011.467298:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763011.467309:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763011.467314:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763011.467324:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2991 00000100:00080000:1.0:1789763011.467336:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763011.467345:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763011.467390:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496192512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763011.467417:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763011.467420:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763011.467422:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763011.467425:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2991 00000100:00080000:1.0:1789763011.467428:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763011.467430:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763011.467440:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763011.467457:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496192640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763011.467472:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763011.467477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763011.467479:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763011.467490:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2991 00000100:00100000:2.0:1789763011.467491:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496192512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763011.467499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763011.467506:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763011.467512:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763011.467522:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496192768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763011.467531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763011.467534:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763011.467536:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763011.467539:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2991 00000100:00080000:1.0:1789763011.467541:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763011.467545:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763011.467548:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763011.467555:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496192896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763011.467615:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496192640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763011.467677:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496192768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763011.467706:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496192896:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763012.043291:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763013.067332:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763014.091481:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763014.539595:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789763014.539691:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763014.539716:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763014.539783:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763014.539802:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763014.539819:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1789763014.539889:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763011/real 1789763014] req@ffff8dd20504c700 x1876699496192640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.539943:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496192640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.539967:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.539977:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763014.539985:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496192640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763014.540000:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763011/real 1789763014] req@ffff8dd20504f800 x1876699496192768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.540050:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496192768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.540067:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.540074:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763014.540081:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496192768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763014.540095:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763011/real 1789763014] req@ffff8dd20504fb80 x1876699496192896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.540121:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496192896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.540137:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.540144:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763014.540150:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496192896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763014.540165:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763011/real 1789763014] req@ffff8dd20504ce00 x1876699496192512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763014.540240:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496192512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763014.540257:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763014.540264:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763014.540270:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496192512:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789763015.115274:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763016.139463:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763016.587359:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763016.587373:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763016.587379:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763016.587392:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 2996 00000100:00080000:0.0:1789763016.587406:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763016.587416:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763016.587499:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496193024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763016.587553:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763016.587560:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763016.587563:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763016.587569:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2996 00000100:00080000:0.0:1789763016.587580:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763016.587586:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763016.587601:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763016.587663:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496193152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763016.587701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496193024:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763016.587741:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763016.587749:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763016.587754:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763016.587766:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 2996 00000100:00080000:0.0:1789763016.587775:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763016.587785:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763016.587799:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763016.587840:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496193280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763016.587863:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763016.587871:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763016.587886:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763016.587894:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 2996 00000100:00080000:0.0:1789763016.587903:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763016.587915:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763016.587924:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763016.587941:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763016.587971:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496193408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763016.588028:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763016/real 1789763016] req@ffff8dd211834700 x1876699496193024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763016.588066:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496193024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763016.588102:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763016.588110:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763016.588117:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496193024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763016.588162:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496193152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763016.588247:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763016.588272:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763016/real 1789763016] req@ffff8dd211835c00 x1876699496193152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763016.588295:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496193152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763016.588309:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763016.588315:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763016.588321:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496193152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763016.588332:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496193280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763016.588370:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763016.588391:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763016/real 1789763016] req@ffff8dd211836a00 x1876699496193280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763016.588447:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496193280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763016.588473:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763016.588480:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763016.588486:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496193280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763016.588499:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496193408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763016.588557:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763016.588579:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763016/real 1789763016] req@ffff8dd211834000 x1876699496193408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763016.588631:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496193408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763016.588641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763016.588647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763016.588651:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496193408:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763017.163422:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763018.187442:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763019.211430:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763020.235411:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763021.259422:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763021.707338:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763021.707360:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763021.707368:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763021.707385:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3001 00000100:00080000:0.0:1789763021.707404:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763021.707416:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763021.707485:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496193536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763021.707538:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763021.707547:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763021.707551:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763021.707560:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3001 00000100:00080000:0.0:1789763021.707590:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763021.707609:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763021.707624:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496193536:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789763021.707626:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763021.707658:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496193664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763021.707675:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763021.707681:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763021.707684:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763021.707689:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3001 00000100:00080000:0.0:1789763021.707694:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763021.707699:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763021.707706:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763021.707721:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496193792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763021.707744:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763021.707750:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763021.707753:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789763021.707762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763021.707765:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3001 00000100:00080000:0.0:1789763021.707774:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763021.707780:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763021.707787:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763021.707803:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496193920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763021.707845:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763021/real 1789763021] req@ffff8dd211835f80 x1876699496193536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763021.707873:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496193536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763021.707890:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763021.707895:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763021.707912:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496193536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763021.707948:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496193664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763021.707983:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763021.707999:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763021/real 1789763021] req@ffff8dd211834380 x1876699496193664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763021.708016:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496193664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763021.708028:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763021.708032:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763021.708036:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496193664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763021.708043:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496193792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763021.708065:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763021.708077:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763021/real 1789763021] req@ffff8dd211836680 x1876699496193792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763021.708092:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496193792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763021.708112:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763021.708116:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763021.708120:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496193792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763021.708128:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496193920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763021.708147:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763021.708159:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763021/real 1789763021] req@ffff8dd211834a80 x1876699496193920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763021.708218:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496193920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763021.708229:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763021.708233:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763021.708238:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496193920:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763022.283374:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763023.307417:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763024.331370:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763025.355419:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763026.379402:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763026.827408:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763026.827424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763026.827431:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763026.827448:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3006 00000100:00080000:3.0:1789763026.827467:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763026.827478:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763026.827567:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496194048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763026.827612:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763026.827617:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763026.827620:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763026.827626:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3006 00000100:00080000:3.0:1789763026.827630:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763026.827637:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763026.827653:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763026.827674:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496194176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763026.827687:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496194048:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763026.827694:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763026.827702:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763026.827705:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763026.827711:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3006 00000100:00080000:3.0:1789763026.827714:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763026.827719:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763026.827727:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763026.827741:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496194304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763026.827755:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763026.827760:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763026.827762:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763026.827767:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3006 00000100:00080000:3.0:1789763026.827784:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763026.827790:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763026.827796:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763026.827804:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763026.827816:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496194432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763026.827882:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763026/real 1789763026] req@ffff8dd203147480 x1876699496194048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763026.827921:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496194048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763026.827937:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763026.827943:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763026.827949:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496194048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763026.827986:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496194176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763026.828019:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763026.828037:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763026/real 1789763026] req@ffff8dd203145880 x1876699496194176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763026.828064:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496194176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763026.828081:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763026.828089:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763026.828096:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496194176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763026.828109:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496194304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763026.828153:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763026.828223:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763026/real 1789763026] req@ffff8dd203147100 x1876699496194304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763026.828241:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496194304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763026.828253:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763026.828259:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763026.828266:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496194304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763026.828279:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496194432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763026.828320:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763026.828345:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763026/real 1789763026] req@ffff8dd203144000 x1876699496194432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763026.828381:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496194432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763026.828392:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763026.828396:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763026.828401:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496194432:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763027.403396:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763028.427405:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763029.451388:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763030.475425:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763031.499415:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763031.947329:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763031.947342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763031.947348:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763031.947361:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3011 00000100:00080000:3.0:1789763031.947374:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763031.947383:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763031.947429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496194560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763031.947470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763031.947475:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763031.947478:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763031.947483:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3011 00000100:00080000:3.0:1789763031.947487:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763031.947517:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763031.947541:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763031.947561:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496194688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763031.947579:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763031.947581:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496194560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763031.947587:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763031.947595:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763031.947606:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3011 00000100:00080000:3.0:1789763031.947614:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763031.947622:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763031.947634:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763031.947662:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496194816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763031.947682:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763031.947689:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763031.947699:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763031.947704:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763031.947715:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3011 00000100:00080000:3.0:1789763031.947724:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763031.947731:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763031.947741:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763031.947744:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763031/real 1789763031] req@ffff8dd203145c00 x1876699496194560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763031.947766:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496194944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763031.947779:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496194560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763031.947801:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763031.947808:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763031.947818:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496194560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763031.947859:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496194688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763031.947908:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763031.947934:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763031/real 1789763031] req@ffff8dd203145180 x1876699496194688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763031.947962:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496194688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763031.947974:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763031.947978:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763031.947982:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496194688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763031.947990:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496194816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763031.948038:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763031.948053:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763031/real 1789763031] req@ffff8dd203147800 x1876699496194816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763031.948078:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496194816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763031.948088:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763031.948092:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763031.948096:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496194816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763031.948103:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496194944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763031.948122:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763031.948134:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763031/real 1789763031] req@ffff8dd203146300 x1876699496194944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763031.948156:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496194944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763031.948230:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763031.948237:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763031.948244:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496194944:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763032.523423:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763033.547424:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763034.571384:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763035.595375:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763036.619300:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763037.067251:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763037.067262:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763037.067266:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763037.067278:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3016 00000100:00080000:3.0:1789763037.067290:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763037.067297:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763037.067338:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496195072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763037.067363:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763037.067366:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763037.067368:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763037.067371:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3016 00000100:00080000:3.0:1789763037.067373:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763037.067376:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763037.067387:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763037.067396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496195200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763037.067404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763037.067407:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763037.067409:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763037.067412:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3016 00000100:00080000:3.0:1789763037.067414:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763037.067417:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763037.067422:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763037.067429:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496195328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763037.067438:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763037.067441:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763037.067443:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763037.067449:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3016 00000100:00100000:2.0:1789763037.067450:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496195072:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763037.067466:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763037.067473:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763037.067478:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763037.067491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496195456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763037.067543:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763037.067586:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763037/real 1789763037] req@ffff8dd203144700 x1876699496195072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763037.067600:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496195072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763037.067608:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763037.067611:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763037.067615:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496195072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763037.067620:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496195200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763037.067637:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763037.067645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763037/real 1789763037] req@ffff8dd203145f80 x1876699496195200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763037.067653:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496195200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763037.067658:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763037.067662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763037.067664:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496195200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763037.067683:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496195328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763037.067695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763037.067702:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763037/real 1789763037] req@ffff8dd203144a80 x1876699496195328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763037.067709:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496195328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763037.067715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763037.067717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763037.067719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496195328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763037.067723:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496195456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763037.067733:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763037.067738:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763037/real 1789763037] req@ffff8dd203145500 x1876699496195456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763037.067745:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496195456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763037.067750:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763037.067752:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763037.067754:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496195456:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763037.643481:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763038.667460:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763039.691412:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763040.715378:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763041.739400:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763042.187266:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763042.187276:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763042.187281:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763042.187290:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3021 00000100:00080000:0.0:1789763042.187326:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763042.187334:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763042.187375:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496195584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763042.187407:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763042.187413:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763042.187433:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763042.187455:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3021 00000100:00080000:0.0:1789763042.187460:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763042.187464:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763042.187472:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763042.187476:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496195584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763042.187484:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496195712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763042.187494:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763042.187497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763042.187498:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763042.187502:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3021 00000100:00080000:0.0:1789763042.187505:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763042.187508:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763042.187513:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763042.187521:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496195840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763042.187530:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763042.187533:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789763042.187533:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763042.187535:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763042.187539:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3021 00000100:00080000:0.0:1789763042.187542:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763042.187546:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763042.187550:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763042.187558:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496195968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763042.187572:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763042/real 1789763042] req@ffff8dd211836d80 x1876699496195584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763042.187589:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496195584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763042.187601:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763042.187605:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763042.187608:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496195584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763042.187627:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496195712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763042.187646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763042.187655:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763042/real 1789763042] req@ffff8dd211835180 x1876699496195712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763042.187663:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496195712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763042.187669:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763042.187671:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763042.187673:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496195712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763042.187677:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496195840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763042.187687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763042.187693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763042/real 1789763042] req@ffff8dd211837480 x1876699496195840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763042.187701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496195840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763042.187706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763042.187707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763042.187709:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496195840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763042.187712:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496195968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763042.187722:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763042.187736:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763042/real 1789763042] req@ffff8dd211834e00 x1876699496195968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763042.187744:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496195968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763042.187749:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763042.187750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763042.187752:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496195968:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763042.763380:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763043.787521:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763044.811298:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763045.835351:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763046.859470:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763047.307331:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763047.307347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763047.307356:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763047.307374:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3026 00000100:00080000:3.0:1789763047.307413:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763047.307429:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763047.307507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496196096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763047.307557:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763047.307562:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763047.307565:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763047.307571:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3026 00000100:00080000:3.0:1789763047.307577:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763047.307583:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763047.307599:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763047.307622:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496196224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763047.307638:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763047.307643:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763047.307649:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763047.307653:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496196096:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763047.307658:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3026 00000100:00080000:3.0:1789763047.307671:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763047.307676:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763047.307684:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763047.307701:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496196352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763047.307725:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763047.307730:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763047.307733:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763047.307738:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3026 00000100:00080000:3.0:1789763047.307756:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763047.307762:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763047.307768:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763047.307780:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763047.307787:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496196480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763047.307894:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763047/real 1789763047] req@ffff8dd203144380 x1876699496196096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763047.307940:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496196096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763047.307955:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763047.307961:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763047.307967:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496196096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763047.308011:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496196224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763047.308063:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763047.308092:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763047/real 1789763047] req@ffff8dd203146d80 x1876699496196224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763047.308114:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496196224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763047.308126:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763047.308131:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763047.308135:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496196224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763047.308143:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496196352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763047.308214:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763047.308233:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763047/real 1789763047] req@ffff8dd203147b80 x1876699496196352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763047.308248:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496196352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763047.308268:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763047.308272:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763047.308276:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496196352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763047.308284:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496196480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763047.308309:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763047.308321:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763047/real 1789763047] req@ffff8dd203146a00 x1876699496196480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763047.308335:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496196480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763047.308345:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763047.308349:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763047.308352:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496196480:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763047.883451:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763048.907409:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763049.931418:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763050.955457:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763051.979413:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763052.427381:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763052.427394:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763052.427400:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763052.427414:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3032 00000100:00080000:0.0:1789763052.427428:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763052.427438:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763052.427487:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496196608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763052.427526:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763052.427531:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763052.427533:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763052.427539:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3032 00000100:00080000:0.0:1789763052.427543:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763052.427547:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763052.427563:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763052.427590:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496196608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763052.427612:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496196736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763052.427640:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763052.427647:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763052.427651:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763052.427659:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3032 00000100:00080000:0.0:1789763052.427667:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763052.427678:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1789763052.427679:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763052.427692:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763052.427716:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496196864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763052.427733:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763052.427737:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789763052.427743:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763052/real 1789763052] req@ffff8dd211834e00 x1876699496196608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763052.427744:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763052.427762:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3032 00000100:00080000:0.0:1789763052.427772:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763052.427780:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763052.427787:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763052.427791:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496196608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763052.427808:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496196992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763052.427843:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763052.427855:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763052.427864:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496196608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763052.427895:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496196736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763052.427931:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763052.427956:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763052/real 1789763052] req@ffff8dd211837480 x1876699496196736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763052.427983:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496196736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763052.428002:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763052.428009:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763052.428016:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496196736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763052.428029:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496196864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763052.428060:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763052.428075:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763052/real 1789763052] req@ffff8dd211835180 x1876699496196864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763052.428090:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496196864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763052.428100:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763052.428104:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763052.428108:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496196864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763052.428116:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496196992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763052.428135:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763052.428147:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763052/real 1789763052] req@ffff8dd211836d80 x1876699496196992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763052.428161:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496196992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763052.428204:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763052.428208:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763052.428211:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496196992:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763053.003445:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763054.027514:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763055.051534:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763056.075424:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763057.099381:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763057.547459:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763057.547476:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763057.547483:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763057.547501:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3037 00000100:00080000:1.0:1789763057.547520:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763057.547531:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763057.547597:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496197120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763057.547647:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763057.547656:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763057.547659:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763057.547665:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3037 00000100:00080000:1.0:1789763057.547669:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763057.547674:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763057.547690:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763057.547707:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496197248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763057.547724:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763057.547729:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763057.547731:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763057.547739:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3037 00000100:00080000:1.0:1789763057.547743:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763057.547752:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496197120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763057.547753:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763057.547765:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763057.547787:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496197376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763057.547802:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763057.547807:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763057.547810:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763057.547817:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3037 00000100:00080000:1.0:1789763057.547822:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763057.547826:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763057.547833:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763057.547849:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496197504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763057.547896:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763057.547957:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763057/real 1789763057] req@ffff8dd20504d180 x1876699496197120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763057.547982:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496197120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763057.547997:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763057.548003:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763057.548008:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496197120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763057.548054:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496197248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763057.548090:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763057.548106:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763057/real 1789763057] req@ffff8dd20504e300 x1876699496197248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763057.548122:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496197248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763057.548133:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763057.548138:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763057.548142:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496197248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763057.548149:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496197376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763057.548205:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763057.548221:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763057/real 1789763057] req@ffff8dd20504d500 x1876699496197376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763057.548236:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496197376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763057.548247:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763057.548251:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763057.548255:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496197376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763057.548262:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496197504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763057.548285:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763057.548298:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763057/real 1789763057] req@ffff8dd20504df80 x1876699496197504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763057.548346:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496197504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763057.548357:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763057.548361:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763057.548364:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496197504:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763058.123322:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763059.147463:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763060.171428:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763061.195429:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763062.219303:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763062.667364:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763062.667378:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763062.667383:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763062.667396:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3042 00000100:00080000:3.0:1789763062.667410:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763062.667421:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763062.667472:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496197632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763062.667514:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763062.667519:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763062.667521:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763062.667527:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3042 00000100:00080000:3.0:1789763062.667531:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763062.667535:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763062.667549:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763062.667571:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496197760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763062.667590:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763062.667600:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789763062.667604:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496197632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763062.667605:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763062.667632:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3042 00000100:00080000:3.0:1789763062.667637:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763062.667641:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763062.667648:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763062.667670:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496197888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763062.667684:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763062.667688:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763062.667691:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763062.667699:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3042 00000100:00080000:3.0:1789763062.667731:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763062.667738:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763062.667745:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763062.667758:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763062.667768:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496198016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763062.667857:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763062/real 1789763062] req@ffff8dd203144e00 x1876699496197632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763062.667900:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496197632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763062.667926:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763062.667937:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763062.667943:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496197632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763062.667988:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496197760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763062.668041:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763062.668068:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763062/real 1789763062] req@ffff8dd203146a00 x1876699496197760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763062.668096:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496197760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763062.668113:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763062.668117:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763062.668122:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496197760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763062.668131:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496197888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763062.668164:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763062.668223:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763062/real 1789763062] req@ffff8dd203147b80 x1876699496197888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763062.668239:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496197888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763062.668250:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763062.668255:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763062.668259:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496197888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763062.668297:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496198016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763062.668332:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763062.668348:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763062/real 1789763062] req@ffff8dd203146d80 x1876699496198016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763062.668363:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496198016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763062.668374:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763062.668378:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763062.668384:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496198016:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763063.243437:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763064.267474:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763065.291496:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763066.315414:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763067.339441:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763067.787349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763067.787362:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763067.787367:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763067.787380:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3047 00000100:00080000:0.0:1789763067.787393:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763067.787401:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763067.787444:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496198144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763067.787498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763067.787502:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763067.787504:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763067.787508:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3047 00000100:00080000:0.0:1789763067.787512:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763067.787516:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763067.787544:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763067.787560:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496198272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763067.787575:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763067.787580:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763067.787582:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763067.787590:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496198144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763067.787592:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3047 00000100:00080000:0.0:1789763067.787606:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763067.787613:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763067.787619:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763067.787635:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496198400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763067.787648:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763067.787652:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763067.787655:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763067.787660:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3047 00000100:00080000:0.0:1789763067.787665:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763067.787670:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763067.787676:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763067.787688:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763067.787693:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496198528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763067.787750:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763067/real 1789763067] req@ffff8dd211834a80 x1876699496198144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763067.787774:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496198144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763067.787788:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763067.787793:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763067.787798:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496198144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763067.787825:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496198272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763067.787853:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763067.787868:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763067/real 1789763067] req@ffff8dd211836680 x1876699496198272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763067.787883:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496198272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763067.787893:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763067.787898:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763067.787902:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496198272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763067.787911:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496198400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763067.787931:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763067.787942:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763067/real 1789763067] req@ffff8dd211834380 x1876699496198400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763067.787956:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496198400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763067.787979:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763067.788000:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763067.788004:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496198400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763067.788012:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496198528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763067.788050:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763067.788072:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763067/real 1789763067] req@ffff8dd211835f80 x1876699496198528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763067.788098:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496198528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763067.788115:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763067.788122:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763067.788129:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496198528:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763068.363459:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763069.387354:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763070.411462:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763071.435409:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763072.459434:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763072.907276:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763072.907284:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763072.907288:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763072.907296:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3052 00000100:00080000:3.0:1789763072.907306:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763072.907312:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763072.907342:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496198656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763072.907363:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763072.907366:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763072.907367:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763072.907370:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3052 00000100:00080000:3.0:1789763072.907373:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763072.907376:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763072.907383:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763072.907396:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496198656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763072.907396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496198784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763072.907409:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763072.907412:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763072.907413:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763072.907415:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3052 00000100:00080000:3.0:1789763072.907418:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763072.907420:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763072.907423:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763072.907431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496198912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763072.907443:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763072.907445:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763072.907446:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763072.907448:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3052 00000100:00080000:3.0:1789763072.907450:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763072.907453:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763072.907456:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763072.907466:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496199040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763072.907468:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763072.907514:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763072/real 1789763072] req@ffff8dd203144380 x1876699496198656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763072.907534:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496198656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763072.907542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763072.907546:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763072.907548:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496198656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763072.907567:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496198784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763072.907586:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763072.907596:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763072/real 1789763072] req@ffff8dd203145500 x1876699496198784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763072.907604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496198784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763072.907609:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763072.907611:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763072.907614:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496198784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763072.907617:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496198912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763072.907628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763072.907634:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763072/real 1789763072] req@ffff8dd203144a80 x1876699496198912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763072.907645:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496198912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763072.907651:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763072.907652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763072.907654:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496198912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763072.907658:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496199040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763072.907668:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763072.907674:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763072/real 1789763072] req@ffff8dd203145f80 x1876699496199040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763072.907681:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496199040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763072.907686:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763072.907687:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763072.907689:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496199040:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763073.483462:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763074.507420:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763075.531465:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763076.555414:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763077.579469:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763078.027302:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763078.027314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763078.027319:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763078.027331:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3057 00000100:00080000:0.0:1789763078.027344:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763078.027353:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763078.027397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496199168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763078.027433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763078.027438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763078.027440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763078.027445:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3057 00000100:00080000:0.0:1789763078.027449:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763078.027453:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763078.027465:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763078.027479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496199296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763078.027493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763078.027499:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763078.027501:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763078.027505:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3057 00000100:00080000:0.0:1789763078.027508:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763078.027514:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763078.027520:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763078.027538:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496199424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763078.027556:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763078.027561:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789763078.027564:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496199168:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763078.027565:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763078.027577:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3057 00000100:00080000:0.0:1789763078.027591:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763078.027603:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763078.027608:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763078.027618:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496199552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763078.027840:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496199296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763078.027922:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496199424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763078.027955:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496199552:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763078.603328:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763079.627391:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763080.651409:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763081.099670:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1789763081.099730:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763081.099755:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763081.099815:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763081.099830:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763081.099846:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789763081.099886:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763078/real 1789763081] req@ffff8dd211835c00 x1876699496199424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763081.099925:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496199424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763081.099942:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763081.099950:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763081.099957:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496199424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763081.099968:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763078/real 1789763081] req@ffff8dd211834700 x1876699496199552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763081.099988:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496199552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763081.100001:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763081.100008:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763081.100013:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496199552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763081.100024:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763078/real 1789763081] req@ffff8dd211834000 x1876699496199168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763081.100045:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496199168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763081.100059:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763081.100064:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763081.100070:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496199168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763081.100081:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763078/real 1789763081] req@ffff8dd211836a00 x1876699496199296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763081.100101:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496199296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763081.100112:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763081.100116:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763081.100121:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496199296:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763081.675282:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763082.699318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763083.147319:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763083.147332:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763083.147338:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763083.147356:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3062 00000100:00080000:2.0:1789763083.147375:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763083.147390:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763083.147449:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496199680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763083.147492:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763083.147497:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763083.147500:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763083.147506:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3062 00000100:00080000:2.0:1789763083.147512:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763083.147518:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763083.147532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763083.147554:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496199808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763083.147575:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763083.147577:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496199680:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763083.147589:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763083.147598:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763083.147608:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3062 00000100:00080000:2.0:1789763083.147616:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763083.147624:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763083.147637:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763083.147667:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496199936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763083.147694:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789763083.147694:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763083.147709:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763083.147714:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763083.147725:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3062 00000100:00080000:2.0:1789763083.147735:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763083.147753:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763083.147764:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763083.147774:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763083/real 1789763083] req@ffff8dd205264700 x1876699496199680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763083.147798:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496200064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763083.147824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496199680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763083.147850:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763083.147872:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763083.147881:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496199680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763083.147923:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496199808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763083.147974:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763083.148002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763083/real 1789763083] req@ffff8dd205266a00 x1876699496199808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763083.148028:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496199808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763083.148047:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763083.148054:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763083.148060:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496199808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763083.148073:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496199936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763083.148147:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763083.148216:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763083/real 1789763083] req@ffff8dd205265880 x1876699496199936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763083.148243:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496199936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763083.148259:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763083.148265:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763083.148271:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496199936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763083.148284:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496200064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763083.148327:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763083.148353:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763083/real 1789763083] req@ffff8dd205264000 x1876699496200064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763083.148378:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496200064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763083.148394:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763083.148400:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763083.148407:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496200064:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763083.723393:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763084.747412:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763085.771384:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763086.795346:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763087.819395:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763088.267285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763088.267297:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763088.267302:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763088.267314:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3067 00000100:00080000:1.0:1789763088.267326:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763088.267334:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763088.267389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496200192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763088.267424:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763088.267430:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763088.267434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763088.267442:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3067 00000100:00080000:1.0:1789763088.267472:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763088.267481:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763088.267494:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496200192:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1789763088.267499:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763088.267537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496200320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763088.267568:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763088.267573:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763088.267576:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763088.267582:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3067 00000100:00080000:1.0:1789763088.267590:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763088.267601:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1789763088.267603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789763088.267614:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763088.267637:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496200448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763088.267651:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763088.267656:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763088.267659:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763088.267667:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763088/real 1789763088] req@ffff8dd20504e680 x1876699496200192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763088.267671:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3067 00000100:00080000:1.0:1789763088.267682:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763088.267694:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763088.267706:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763088.267726:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496200576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763088.267735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496200192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763088.267765:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763088.267775:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763088.267784:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496200192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763088.267839:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496200320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763088.267872:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763088.267888:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763088/real 1789763088] req@ffff8dd20504df80 x1876699496200320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763088.267903:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496200320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763088.267913:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763088.267917:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763088.267921:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496200320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763088.267927:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496200448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763088.267946:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763088.267957:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763088/real 1789763088] req@ffff8dd20504d500 x1876699496200448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763088.267970:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496200448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763088.267979:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763088.267982:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763088.267985:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496200448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763088.267991:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496200576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763088.268007:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763088.268017:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763088/real 1789763088] req@ffff8dd20504e300 x1876699496200576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763088.268029:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496200576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763088.268045:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763088.268049:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763088.268052:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496200576:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763088.843379:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763089.867472:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763090.891356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763091.915376:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763092.939333:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763093.387314:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763093.387328:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763093.387334:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763093.387348:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3072 00000100:00080000:1.0:1789763093.387361:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763093.387370:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763093.387419:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496200704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763093.387458:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763093.387467:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763093.387470:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763093.387475:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3072 00000100:00080000:1.0:1789763093.387480:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763093.387484:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763093.387497:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763093.387515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496200832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763093.387530:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763093.387535:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763093.387538:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763093.387546:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496200704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763093.387548:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3072 00000100:00080000:1.0:1789763093.387558:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763093.387567:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763093.387573:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763093.387588:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496200960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763093.387605:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763093.387613:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763093.387616:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763093.387639:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3072 00000100:00080000:1.0:1789763093.387646:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763093.387654:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1789763093.387658:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789763093.387669:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763093.387694:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496201088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763093.387712:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763093/real 1789763093] req@ffff8dd20504d180 x1876699496200704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763093.387735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496200704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763093.387748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763093.387753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763093.387757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496200704:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763093.387781:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496200832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763093.387804:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763093.387816:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763093/real 1789763093] req@ffff8dd20504ce00 x1876699496200832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763093.387829:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496200832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763093.387837:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763093.387841:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763093.387844:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496200832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763093.387850:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496200960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763093.387867:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763093.387876:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763093/real 1789763093] req@ffff8dd20504fb80 x1876699496200960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763093.387888:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496200960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763093.387896:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763093.387899:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763093.387902:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496200960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763093.387907:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496201088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763093.387922:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763093.387931:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763093/real 1789763093] req@ffff8dd20504f800 x1876699496201088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763093.387942:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496201088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763093.387950:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763093.387953:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763093.387956:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496201088:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763093.963349:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763094.987374:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763096.011641:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763097.035453:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763098.059418:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763098.507299:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763098.507313:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763098.507319:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763098.507332:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3078 00000100:00080000:0.0:1789763098.507345:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763098.507354:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763098.507404:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496201216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763098.507443:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763098.507449:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763098.507452:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763098.507458:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3078 00000100:00080000:0.0:1789763098.507462:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763098.507466:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763098.507479:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763098.507500:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496201344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763098.507521:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763098.507527:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763098.507531:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763098.507539:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3078 00000100:00080000:0.0:1789763098.507546:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763098.507564:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763098.507570:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496201216:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789763098.507578:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763098.507613:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496201472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763098.507637:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763098.507644:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763098.507648:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763098.507656:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3078 00000100:00080000:0.0:1789763098.507665:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763098.507674:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763098.507686:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763098.507714:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496201600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763098.507724:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763098.507830:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763098/real 1789763098] req@ffff8dd211837100 x1876699496201216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763098.507865:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496201216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763098.507882:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763098.507889:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763098.507895:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496201216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763098.507907:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496201344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763098.507944:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763098.507962:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763098/real 1789763098] req@ffff8dd211836300 x1876699496201344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763098.507979:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496201344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763098.507990:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763098.507994:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763098.507999:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496201344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763098.508080:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496201472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763098.508124:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763098.508148:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763098/real 1789763098] req@ffff8dd211837800 x1876699496201472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763098.508224:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496201472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763098.508249:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763098.508254:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763098.508259:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496201472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763098.508268:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496201600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763098.508301:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763098.508317:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763098/real 1789763098] req@ffff8dd211837b80 x1876699496201600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763098.508332:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496201600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763098.508343:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763098.508348:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763098.508352:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496201600:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763099.083296:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763100.107421:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763101.131420:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763102.155376:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763103.179306:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763103.627378:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763103.627399:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763103.627408:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763103.627430:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3083 00000100:00080000:2.0:1789763103.627454:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763103.627470:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763103.627549:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496201728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763103.627605:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763103.627612:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763103.627616:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763103.627633:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3083 00000100:00080000:2.0:1789763103.627640:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763103.627646:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763103.627663:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763103.627682:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496201856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763103.627696:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763103.627700:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763103.627702:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763103.627709:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3083 00000100:00080000:2.0:1789763103.627713:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763103.627718:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763103.627725:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763103.627738:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496201984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763103.627752:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763103.627756:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763103.627758:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763103.627763:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3083 00000100:00080000:2.0:1789763103.627767:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763103.627771:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763103.627777:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763103.627786:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496201728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763103.627812:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496202112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763103.627904:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763103.628001:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763103/real 1789763103] req@ffff8dd205267b80 x1876699496201728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763103.628041:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496201728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763103.628064:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763103.628073:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763103.628082:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496201728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763103.628099:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496201856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763103.628140:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763103.628160:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763103/real 1789763103] req@ffff8dd205267100 x1876699496201856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763103.628232:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496201856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763103.628244:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763103.628248:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763103.628253:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496201856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763103.628261:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496201984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763103.628290:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763103.628304:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763103/real 1789763103] req@ffff8dd205264e00 x1876699496201984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763103.628319:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496201984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763103.628330:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763103.628334:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763103.628338:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496201984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763103.628377:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496202112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763103.628398:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763103.628410:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763103/real 1789763103] req@ffff8dd205264380 x1876699496202112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763103.628426:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496202112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763103.628437:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763103.628440:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763103.628444:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496202112:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763104.203455:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763105.227408:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763106.251432:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763107.275462:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763108.299465:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763108.747307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763108.747329:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763108.747349:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763108.747371:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3088 00000100:00080000:2.0:1789763108.747409:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763108.747424:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763108.747492:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496202240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763108.747537:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763108.747543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763108.747545:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763108.747552:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3088 00000100:00080000:2.0:1789763108.747557:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763108.747562:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763108.747577:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763108.747593:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496202368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763108.747606:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763108.747611:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763108.747613:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763108.747629:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3088 00000100:00080000:2.0:1789763108.747633:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763108.747637:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763108.747642:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763108.747654:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496202496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763108.747667:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763108.747671:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763108.747673:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763108.747678:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3088 00000100:00080000:2.0:1789763108.747683:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763108.747688:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763108.747693:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763108.747704:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496202624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763108.747774:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496202240:192.168.203.109@tcp:250:kworker.0 00000800:00000100:3.0:1789763108.747867:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763108.747908:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763108/real 1789763108] req@ffff8dd205267800 x1876699496202240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763108.747944:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496202240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763108.747956:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763108.747977:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763108.747982:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496202240:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763108.747992:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496202368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763108.748020:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763108.748055:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763108/real 1789763108] req@ffff8dd205264a80 x1876699496202368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763108.748071:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496202368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763108.748090:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763108.748095:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763108.748099:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496202368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763108.748107:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496202496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763108.748129:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763108.748141:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763108/real 1789763108] req@ffff8dd205267480 x1876699496202496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763108.748155:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496202496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763108.748166:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763108.748170:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763108.748173:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496202496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763108.748181:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496202624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763108.748245:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763108.748274:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763108/real 1789763108] req@ffff8dd205266680 x1876699496202624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763108.748283:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496202624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763108.748290:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763108.748293:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763108.748296:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496202624:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763109.323427:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763110.347400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763111.371475:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763112.395309:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763113.419482:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763113.867331:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763113.867344:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763113.867350:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763113.867364:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3093 00000100:00080000:2.0:1789763113.867377:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763113.867388:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763113.867436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496202752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763113.867473:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763113.867478:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763113.867481:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763113.867486:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3093 00000100:00080000:2.0:1789763113.867490:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763113.867494:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763113.867505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763113.867524:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496202880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763113.867538:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763113.867543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763113.867545:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763113.867550:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3093 00000100:00080000:2.0:1789763113.867555:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763113.867564:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496202752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763113.867565:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763113.867581:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763113.867600:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496203008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763113.867615:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763113.867620:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763113.867622:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763113.867628:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3093 00000100:00080000:2.0:1789763113.867645:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763113.867650:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763113.867656:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763113.867666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763113.867675:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496203136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763113.867744:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763113/real 1789763113] req@ffff8dd205265500 x1876699496202752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763113.867783:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496202752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763113.867825:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763113.867835:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763113.867843:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496202752:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763113.867858:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496202880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763113.867924:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763113.867953:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763113/real 1789763113] req@ffff8dd205266680 x1876699496202880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763113.867971:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496202880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763113.867982:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763113.867986:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763113.867991:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496202880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763113.868023:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496203008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763113.868049:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763113.868063:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763113/real 1789763113] req@ffff8dd205267480 x1876699496203008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763113.868098:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496203008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763113.868114:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763113.868120:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763113.868127:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496203008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763113.868138:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496203136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763113.868219:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763113.868238:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763113/real 1789763113] req@ffff8dd205264a80 x1876699496203136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763113.868254:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496203136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763113.868265:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763113.868269:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763113.868273:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496203136:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763114.443412:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763115.467394:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763116.491371:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763117.515395:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763118.539383:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763118.987354:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763118.987366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763118.987371:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763118.987384:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3098 00000100:00080000:1.0:1789763118.987398:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763118.987408:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763118.987454:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496203264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763118.987492:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763118.987497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763118.987500:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763118.987505:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3098 00000100:00080000:1.0:1789763118.987509:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763118.987513:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763118.987524:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763118.987539:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496203392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763118.987553:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763118.987557:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763118.987559:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763118.987565:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3098 00000100:00080000:1.0:1789763118.987570:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763118.987574:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763118.987582:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763118.987594:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496203520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763118.987607:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496203264:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763118.987623:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763118.987636:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763118.987638:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763118.987644:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3098 00000100:00080000:1.0:1789763118.987648:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763118.987651:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763118.987657:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763118.987671:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496203648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763118.987719:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763118.987771:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763118/real 1789763118] req@ffff8dd20504c700 x1876699496203264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763118.987796:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496203264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763118.987811:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763118.987816:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763118.987821:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496203264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763118.987831:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496203392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763118.987889:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763118.987914:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763118/real 1789763118] req@ffff8dd20504f100 x1876699496203392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763118.987930:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496203392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763118.987941:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763118.987946:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763118.987950:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496203392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763118.987981:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496203520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763118.988006:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763118.988019:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763118/real 1789763118] req@ffff8dd20504ed80 x1876699496203520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763118.988033:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496203520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763118.988044:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763118.988047:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763118.988051:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496203520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763118.988058:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496203648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763118.988078:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763118.988090:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763118/real 1789763118] req@ffff8dd20504d880 x1876699496203648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763118.988104:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496203648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763118.988113:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763118.988117:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763118.988120:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496203648:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763119.563437:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763120.587435:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763121.611405:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763122.635407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763123.659443:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763124.107326:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763124.107339:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763124.107346:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763124.107359:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3103 00000100:00080000:2.0:1789763124.107374:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763124.107385:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763124.107436:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496203776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763124.107485:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763124.107493:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763124.107497:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763124.107506:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3103 00000100:00080000:2.0:1789763124.107514:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763124.107524:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763124.107539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763124.107554:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496203776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763124.107564:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496203904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763124.107585:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763124.107589:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763124.107592:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763124.107597:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3103 00000100:00080000:2.0:1789763124.107602:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763124.107606:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763124.107614:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763124.107630:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496204032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763124.107644:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763124.107648:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763124.107650:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763124.107657:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3103 00000100:00080000:2.0:1789763124.107664:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763124.107672:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1789763124.107682:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789763124.107684:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763124.107713:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496204160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763124.107744:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763124/real 1789763124] req@ffff8dd205267800 x1876699496203776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763124.107774:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496203776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763124.107788:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763124.107793:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763124.107798:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496203776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763124.107861:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496203904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763124.107909:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763124.107935:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763124/real 1789763124] req@ffff8dd205264380 x1876699496203904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763124.107958:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496203904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763124.107969:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763124.107973:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763124.107978:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496203904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763124.107986:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496204032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763124.108014:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763124.108027:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763124/real 1789763124] req@ffff8dd205264e00 x1876699496204032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763124.108042:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496204032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763124.108052:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763124.108056:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763124.108060:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496204032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763124.108066:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496204160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763124.108085:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763124.108096:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763124/real 1789763124] req@ffff8dd205267100 x1876699496204160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763124.108110:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496204160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763124.108128:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763124.108132:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763124.108136:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496204160:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763124.683431:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763125.707426:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763126.731423:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763127.755469:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763128.779434:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763129.227240:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763129.227255:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763129.227259:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763129.227283:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3108 00000100:00080000:2.0:1789763129.227294:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763129.227300:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763129.227333:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496204288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763129.227356:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763129.227359:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763129.227362:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763129.227380:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3108 00000100:00080000:2.0:1789763129.227383:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763129.227386:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763129.227395:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763129.227411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496204416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763129.227421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763129.227423:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496204288:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763129.227424:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763129.227429:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763129.227435:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3108 00000100:00080000:2.0:1789763129.227437:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763129.227440:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763129.227444:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763129.227454:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496204544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763129.227463:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763129.227465:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763129.227467:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763129.227470:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3108 00000100:00080000:2.0:1789763129.227476:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763129.227479:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763129.227483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763129.227490:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763129.227495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496204672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763129.227532:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763129/real 1789763129] req@ffff8dd205267b80 x1876699496204288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763129.227549:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496204288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763129.227557:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763129.227576:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763129.227579:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496204288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763129.227597:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496204416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763129.227614:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763129.227622:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763129/real 1789763129] req@ffff8dd205264000 x1876699496204416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763129.227646:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496204416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763129.227653:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763129.227656:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763129.227658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496204416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763129.227663:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496204544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763129.227677:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763129.227684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763129/real 1789763129] req@ffff8dd205265880 x1876699496204544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763129.227692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496204544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763129.227698:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763129.227700:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763129.227702:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496204544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763129.227706:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496204672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763129.227717:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763129.227723:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763129/real 1789763129] req@ffff8dd205266a00 x1876699496204672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763129.227731:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496204672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763129.227737:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763129.227739:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763129.227741:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496204672:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763129.803399:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763130.827407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763131.851377:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763132.875400:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763133.899439:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763134.347288:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763134.347302:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763134.347308:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763134.347322:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3113 00000100:00080000:2.0:1789763134.347337:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763134.347347:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763134.347394:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496204800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763134.347431:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763134.347435:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763134.347438:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763134.347443:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3113 00000100:00080000:2.0:1789763134.347447:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763134.347451:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763134.347463:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763134.347479:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496204928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763134.347493:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763134.347498:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763134.347503:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763134.347513:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3113 00000100:00100000:3.0:1789763134.347517:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496204800:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763134.347519:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763134.347531:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763134.347540:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763134.347555:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496205056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763134.347571:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763134.347576:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763134.347577:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763134.347583:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3113 00000100:00080000:2.0:1789763134.347600:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763134.347605:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763134.347611:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763134.347619:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763134.347634:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496205184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763134.347680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763134/real 1789763134] req@ffff8dd205264700 x1876699496204800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763134.347710:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496204800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763134.347724:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763134.347730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763134.347735:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496204800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763134.347772:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496204928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763134.347813:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763134.347839:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763134/real 1789763134] req@ffff8dd205266300 x1876699496204928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763134.347866:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496204928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763134.347884:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763134.347889:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763134.347894:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496204928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763134.347939:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496205056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763134.347971:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763134.347986:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763134/real 1789763134] req@ffff8dd205265180 x1876699496205056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763134.348002:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496205056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763134.348020:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763134.348024:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763134.348028:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496205056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763134.348035:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496205184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763134.348062:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763134.348081:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763134/real 1789763134] req@ffff8dd205266d80 x1876699496205184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763134.348105:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496205184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763134.348122:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763134.348128:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763134.348135:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496205184:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763134.923369:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763135.947363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763136.971361:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763137.995451:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763139.019366:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763139.467335:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763139.467361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763139.467367:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763139.467380:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3119 00000100:00080000:0.0:1789763139.467393:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763139.467402:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763139.467451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496205312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763139.467490:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763139.467497:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763139.467501:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763139.467509:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3119 00000100:00080000:0.0:1789763139.467516:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763139.467524:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763139.467539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763139.467572:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496205312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763139.467587:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496205440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763139.467606:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763139.467612:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763139.467616:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763139.467622:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3119 00000100:00080000:0.0:1789763139.467628:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763139.467635:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763139.467644:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763139.467662:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496205568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763139.467662:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763139.467679:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763139.467685:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763139.467688:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763139.467695:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3119 00000100:00080000:0.0:1789763139.467728:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763139.467736:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763139.467745:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763139.467750:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763139/real 1789763139] req@ffff8dd211837b80 x1876699496205312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763139.467766:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496205696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763139.467784:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496205312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763139.467803:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763139.467811:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763139.467818:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496205312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763139.467859:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496205440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763139.467897:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763139.467915:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763139/real 1789763139] req@ffff8dd211837800 x1876699496205440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763139.467937:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496205440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763139.467954:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763139.467960:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763139.467966:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496205440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763139.467978:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496205568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763139.468037:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763139.468081:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763139/real 1789763139] req@ffff8dd211836300 x1876699496205568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763139.468120:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496205568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763139.468133:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763139.468138:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763139.468143:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496205568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763139.468152:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496205696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763139.468182:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763139.468239:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763139/real 1789763139] req@ffff8dd211837100 x1876699496205696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763139.468259:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496205696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763139.468272:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763139.468277:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763139.468282:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496205696:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763140.043323:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763141.067438:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763142.091388:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763143.115375:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763144.139371:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763144.587340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763144.587358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763144.587366:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763144.587384:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3124 00000100:00080000:2.0:1789763144.587402:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763144.587413:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763144.587463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496205824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763144.587507:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763144.587525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763144.587528:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763144.587538:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3124 00000100:00080000:2.0:1789763144.587547:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763144.587556:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763144.587574:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763144.587587:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496205824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763144.587606:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496205952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763144.587625:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763144.587630:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763144.587633:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763144.587638:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3124 00000100:00080000:2.0:1789763144.587658:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763144.587691:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763144.587700:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763144.587719:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496206080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763144.587737:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763144.587742:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763144.587747:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763144.587753:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3124 00000100:00080000:2.0:1789763144.587759:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763144.587766:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496205952:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1789763144.587771:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763144.587786:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763144.587809:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496206208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763144.587837:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496206080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763144.587886:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496206208:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763145.163355:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763146.187441:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763147.211459:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763147.659480:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789763147.659521:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763147.659537:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763147.659600:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763147.659636:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763147.659658:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789763147.659747:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763144/real 1789763147] req@ffff8dd205266300 x1876699496206208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763147.659775:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496206208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763147.659791:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763147.659797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763147.659803:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496206208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763147.659814:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763144/real 1789763147] req@ffff8dd205266d80 x1876699496205952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763147.659830:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496205952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763147.659841:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763147.659846:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763147.659850:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496205952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763147.659858:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763144/real 1789763147] req@ffff8dd205265180 x1876699496206080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763147.659874:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496206080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763147.659893:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763147.659898:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763147.659903:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496206080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763147.659912:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763144/real 1789763147] req@ffff8dd205265f80 x1876699496205824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763147.659938:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496205824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763147.659949:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763147.659954:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763147.659957:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496205824:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789763148.235377:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763149.259505:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763149.707288:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763149.707301:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763149.707306:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763149.707320:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3129 00000100:00080000:2.0:1789763149.707336:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763149.707347:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763149.707394:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496206336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763149.707450:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763149.707454:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763149.707456:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763149.707462:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3129 00000100:00080000:2.0:1789763149.707468:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763149.707472:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763149.707482:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763149.707499:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496206464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763149.707522:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763149.707527:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763149.707529:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763149.707534:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3129 00000100:00080000:2.0:1789763149.707538:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763149.707554:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763149.707557:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496206336:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789763149.707568:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763149.707592:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496206592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763149.707612:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763149.707648:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763149.707650:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763149.707656:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3129 00000100:00080000:2.0:1789763149.707673:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763149.707678:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763149.707684:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763149.707698:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496206720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763149.707725:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763149.707804:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763149/real 1789763149] req@ffff8dd205264700 x1876699496206336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763149.707842:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496206336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763149.707863:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763149.707871:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763149.707878:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496206336:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763149.707892:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496206464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763149.707939:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763149.707981:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763149/real 1789763149] req@ffff8dd205266a00 x1876699496206464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763149.708012:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496206464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763149.708028:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763149.708034:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763149.708041:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496206464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763149.708099:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496206592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763149.708152:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763149.708217:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763149/real 1789763149] req@ffff8dd205265880 x1876699496206592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763149.708242:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496206592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763149.708268:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763149.708274:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763149.708281:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496206592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763149.708293:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496206720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763149.708892:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763149.708934:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763149/real 1789763149] req@ffff8dd205264000 x1876699496206720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763149.708961:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496206720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763149.708988:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763149.708998:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763149.709004:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496206720:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763150.283416:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763151.307420:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763152.331397:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763153.355412:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763154.379384:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763154.827307:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763154.827320:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763154.827327:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763154.827341:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3134 00000100:00080000:2.0:1789763154.827356:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763154.827367:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763154.827416:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496206848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763154.827454:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763154.827460:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763154.827462:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763154.827467:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3134 00000100:00080000:2.0:1789763154.827473:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763154.827478:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763154.827490:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763154.827506:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496206976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763154.827520:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763154.827526:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763154.827529:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763154.827533:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3134 00000100:00080000:2.0:1789763154.827538:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763154.827546:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763154.827559:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763154.827559:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496206848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763154.827617:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496207104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763154.827646:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763154.827651:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763154.827654:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763154.827661:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3134 00000100:00080000:2.0:1789763154.827665:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763154.827671:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763154.827678:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763154.827695:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496207232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763154.827741:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763154.827811:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763154/real 1789763154] req@ffff8dd205267b80 x1876699496206848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763154.827840:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496206848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763154.827857:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763154.827864:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763154.827870:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496206848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763154.827883:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496206976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763154.827915:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763154.827932:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763154/real 1789763154] req@ffff8dd205267100 x1876699496206976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763154.827949:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496206976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763154.827960:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763154.827964:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763154.827968:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496206976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763154.828005:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496207104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763154.828030:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763154.828058:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763154/real 1789763154] req@ffff8dd205264e00 x1876699496207104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763154.828091:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496207104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763154.828110:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763154.828118:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763154.828126:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496207104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763154.828139:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496207232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763154.828227:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763154.828255:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763154/real 1789763154] req@ffff8dd205264380 x1876699496207232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763154.828285:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496207232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763154.828303:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763154.828310:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763154.828315:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496207232:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763155.403492:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763156.427475:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763157.451442:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763158.475323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763159.499424:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763159.947349:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763159.947362:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763159.947367:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763159.947379:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3139 00000100:00080000:0.0:1789763159.947394:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763159.947402:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763159.947464:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496207360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763159.947505:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763159.947513:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763159.947517:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763159.947525:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3139 00000100:00080000:0.0:1789763159.947532:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763159.947540:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763159.947581:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763159.947608:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496207488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763159.947623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496207360:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763159.947628:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763159.947635:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763159.947638:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763159.947645:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3139 00000100:00080000:0.0:1789763159.947651:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763159.947657:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763159.947666:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763159.947683:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496207616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763159.947700:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763159.947706:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763159.947709:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763159.947716:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3139 00000800:00000100:3.0:1789763159.947716:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763159.947722:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763159.947728:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763159.947737:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763159.947753:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496207744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763159.947777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763159/real 1789763159] req@ffff8dd211836a00 x1876699496207360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763159.947810:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496207360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763159.947831:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763159.947839:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763159.947846:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496207360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763159.947879:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496207488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763159.947914:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763159.947931:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763159/real 1789763159] req@ffff8dd211834000 x1876699496207488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763159.947947:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496207488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763159.947966:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763159.947970:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763159.947974:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496207488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763159.947981:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496207616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763159.948001:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763159.948012:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763159/real 1789763159] req@ffff8dd211834700 x1876699496207616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763159.948026:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496207616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763159.948036:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763159.948039:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763159.948043:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496207616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763159.948050:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496207744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763159.948069:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763159.948080:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763159/real 1789763159] req@ffff8dd211835c00 x1876699496207744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763159.948094:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496207744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763159.948103:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763159.948106:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763159.948111:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496207744:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763160.523456:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763161.547481:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763162.571423:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763163.595396:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763164.619479:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763165.067326:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763165.067338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763165.067344:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763165.067356:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3144 00000100:00080000:0.0:1789763165.067371:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763165.067381:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763165.067431:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496207872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763165.067465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763165.067470:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763165.067472:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763165.067477:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3144 00000100:00080000:0.0:1789763165.067480:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763165.067485:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763165.067497:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763165.067544:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496208000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763165.067562:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763165.067566:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496207872:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763165.067573:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763165.067582:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763165.067592:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3144 00000100:00080000:0.0:1789763165.067602:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763165.067611:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763165.067624:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763165.067652:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496208128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763165.067664:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763165.067688:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763165.067695:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763165.067699:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763165.067706:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3144 00000100:00080000:0.0:1789763165.067714:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763165.067721:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763165.067730:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763165.067732:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763165/real 1789763165] req@ffff8dd211835f80 x1876699496207872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763165.067755:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496208256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763165.067767:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496207872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763165.067788:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763165.067795:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763165.067802:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496207872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763165.067850:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496208000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763165.067882:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763165.067898:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763165/real 1789763165] req@ffff8dd211834380 x1876699496208000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763165.067914:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496208000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763165.067925:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763165.067929:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763165.067933:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496208000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763165.067940:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496208128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763165.067964:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763165.067985:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763165/real 1789763165] req@ffff8dd211836680 x1876699496208128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763165.068008:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496208128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763165.068026:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763165.068032:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763165.068039:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496208128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763165.068051:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496208256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763165.068091:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763165.068111:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763165/real 1789763165] req@ffff8dd211834a80 x1876699496208256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763165.068127:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496208256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763165.068137:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763165.068141:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763165.068145:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496208256:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763165.643504:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763166.667431:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763167.691330:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763168.715456:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763169.739396:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763170.187447:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763170.187463:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763170.187471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763170.187519:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3149 00000100:00080000:3.0:1789763170.187536:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763170.187547:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763170.187623:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496208384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763170.187671:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763170.187677:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763170.187680:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763170.187686:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3149 00000100:00080000:3.0:1789763170.187690:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763170.187694:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763170.187710:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763170.187727:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496208512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763170.187741:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763170.187747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763170.187749:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763170.187753:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3149 00000100:00080000:3.0:1789763170.187757:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763170.187762:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763170.187773:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763170.187790:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496208384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763170.187790:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496208640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763170.187833:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763170.187839:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763170.187842:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763170.187851:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3149 00000100:00080000:3.0:1789763170.187856:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763170.187862:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763170.187869:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763170.187887:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496208768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763170.187927:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763170.187978:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763170/real 1789763170] req@ffff8dd203144700 x1876699496208384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763170.187993:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496208384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763170.188002:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763170.188006:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763170.188009:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496208384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763170.188016:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496208512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763170.188033:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763170.188041:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763170/real 1789763170] req@ffff8dd203146300 x1876699496208512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763170.188060:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496208512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763170.188066:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763170.188068:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763170.188071:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496208512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763170.188092:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496208640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763170.188106:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763170.188114:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763170/real 1789763170] req@ffff8dd203147800 x1876699496208640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763170.188123:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496208640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763170.188129:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763170.188131:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763170.188133:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496208640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763170.188137:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496208768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763170.188148:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763170.188154:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763170/real 1789763170] req@ffff8dd203145180 x1876699496208768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763170.188213:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496208768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763170.188220:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763170.188222:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763170.188225:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496208768:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763170.763458:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763171.787436:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763172.811429:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763173.835407:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763174.859483:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763175.307274:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763175.307287:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763175.307293:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763175.307306:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3154 00000100:00080000:3.0:1789763175.307320:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763175.307329:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763175.307378:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496208896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763175.307416:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763175.307422:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763175.307426:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763175.307432:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3154 00000100:00080000:3.0:1789763175.307462:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763175.307470:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763175.307483:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763175.307500:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496208896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763175.307507:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496209024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763175.307531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763175.307536:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763175.307539:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763175.307544:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3154 00000100:00080000:3.0:1789763175.307548:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763175.307553:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763175.307560:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763175.307576:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496209152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763175.307590:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763175.307595:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763175.307597:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763175.307607:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3154 00000800:00000100:2.0:1789763175.307609:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763175.307613:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763175.307621:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763175.307629:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763175.307644:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496209280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763175.307675:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763175/real 1789763175] req@ffff8dd203145c00 x1876699496208896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763175.307702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496208896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763175.307717:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763175.307722:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763175.307728:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496208896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763175.307758:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496209024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763175.307786:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763175.307802:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763175/real 1789763175] req@ffff8dd203144000 x1876699496209024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763175.307826:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496209024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763175.307856:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763175.307862:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763175.307869:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496209024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763175.307882:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496209152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763175.307923:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763175.307939:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763175/real 1789763175] req@ffff8dd203147100 x1876699496209152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763175.307971:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496209152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763175.307982:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763175.307986:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763175.307990:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496209152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763175.307997:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496209280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763175.308022:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763175.308035:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763175/real 1789763175] req@ffff8dd203145880 x1876699496209280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763175.308049:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496209280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763175.308059:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763175.308062:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763175.308078:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496209280:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763175.883469:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763176.907470:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763177.931464:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763178.955392:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763179.979355:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763180.427315:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763180.427327:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763180.427332:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763180.427346:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3160 00000100:00080000:3.0:1789763180.427359:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763180.427368:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763180.427443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496209408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763180.427481:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763180.427486:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763180.427489:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763180.427495:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3160 00000100:00080000:3.0:1789763180.427499:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763180.427503:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763180.427515:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763180.427533:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496209536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763180.427549:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763180.427558:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789763180.427561:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496209408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763180.427561:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763180.427572:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3160 00000100:00080000:3.0:1789763180.427582:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763180.427588:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763180.427594:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763180.427620:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496209664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763180.427635:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763180.427640:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763180.427642:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763180.427646:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3160 00000100:00080000:3.0:1789763180.427653:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763180.427658:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763180.427664:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763180.427670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763180.427686:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496209792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763180.427735:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763180/real 1789763180] req@ffff8dd203147480 x1876699496209408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763180.427761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496209408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763180.427775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763180.427780:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763180.427786:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496209408:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763180.427815:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496209536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763180.427843:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763180.427858:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763180/real 1789763180] req@ffff8dd203145880 x1876699496209536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763180.427873:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496209536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763180.427884:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763180.427888:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763180.427892:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496209536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763180.427899:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496209664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763180.427920:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763180.427932:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763180/real 1789763180] req@ffff8dd203147100 x1876699496209664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763180.427946:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496209664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763180.427956:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763180.427960:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763180.427963:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496209664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763180.427970:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496209792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763180.427989:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763180.428000:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763180/real 1789763180] req@ffff8dd203144000 x1876699496209792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763180.428014:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496209792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763180.428024:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763180.428027:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763180.428031:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496209792:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763181.003507:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763182.027465:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763183.051471:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763184.075400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763185.099483:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763185.547313:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763185.547326:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763185.547332:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763185.547347:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3165 00000100:00080000:2.0:1789763185.547394:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763185.547407:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763185.547459:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496209920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763185.547494:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763185.547499:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763185.547502:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763185.547507:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3165 00000100:00080000:2.0:1789763185.547512:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763185.547518:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763185.547530:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763185.547546:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496210048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763185.547559:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496209920:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763185.547566:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763185.547573:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763185.547576:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763185.547582:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3165 00000100:00080000:2.0:1789763185.547585:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763185.547590:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763185.547597:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763185.547614:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496210176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763185.547626:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763185.547630:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763185.547633:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763185.547639:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3165 00000100:00080000:2.0:1789763185.547644:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763185.547650:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1789763185.547658:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789763185.547664:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763185.547683:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496210304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763185.547723:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763185/real 1789763185] req@ffff8dd205267800 x1876699496209920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763185.547750:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496209920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763185.547765:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763185.547770:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763185.547775:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496209920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763185.547804:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496210048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763185.547832:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763185.547862:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763185/real 1789763185] req@ffff8dd205264a80 x1876699496210048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763185.547879:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496210048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763185.547890:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763185.547894:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763185.547900:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496210048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763185.547907:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496210176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763185.547931:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763185.547944:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763185/real 1789763185] req@ffff8dd205267480 x1876699496210176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763185.547958:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496210176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763185.547968:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763185.547972:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763185.547976:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496210176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763185.547983:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496210304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763185.548002:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763185.548012:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763185/real 1789763185] req@ffff8dd205266680 x1876699496210304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763185.548026:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496210304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763185.548036:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763185.548039:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763185.548043:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496210304:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763186.123452:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763187.147449:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763188.171368:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763189.195423:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763190.219386:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763190.667309:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763190.667345:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763190.667351:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763190.667365:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3170 00000100:00080000:2.0:1789763190.667379:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763190.667388:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763190.667435:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496210432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763190.667472:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763190.667477:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763190.667480:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763190.667485:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3170 00000100:00080000:2.0:1789763190.667489:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763190.667493:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763190.667504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763190.667521:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496210560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763190.667535:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763190.667541:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763190.667544:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763190.667550:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3170 00000100:00080000:2.0:1789763190.667554:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763190.667558:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763190.667564:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763190.667576:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496210688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763190.667589:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496210432:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763190.667594:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763190.667607:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763190.667610:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763190.667616:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3170 00000100:00080000:2.0:1789763190.667620:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763190.667623:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763190.667630:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763190.667647:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496210816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763190.667691:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763190.667746:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763190/real 1789763190] req@ffff8dd205265500 x1876699496210432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763190.667770:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496210432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763190.667785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763190.667790:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763190.667795:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496210432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763190.667804:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496210560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763190.667829:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763190.667843:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763190/real 1789763190] req@ffff8dd205266680 x1876699496210560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763190.667858:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496210560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763190.667868:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763190.667872:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763190.667876:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496210560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763190.667904:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496210688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763190.667942:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763190.667955:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763190/real 1789763190] req@ffff8dd205267480 x1876699496210688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763190.667970:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496210688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763190.667989:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763190.667993:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763190.667997:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496210688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763190.668003:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496210816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763190.668023:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763190.668034:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763190/real 1789763190] req@ffff8dd205264a80 x1876699496210816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763190.668048:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496210816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763190.668057:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763190.668061:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763190.668064:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496210816:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763191.243476:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763192.267486:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763193.291445:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763194.315363:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763195.339462:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763195.787259:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763195.787267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763195.787270:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763195.787279:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3175 00000100:00080000:2.0:1789763195.787300:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763195.787306:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763195.787339:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496210944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763195.787361:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763195.787364:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763195.787365:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763195.787368:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3175 00000100:00080000:2.0:1789763195.787370:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763195.787373:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763195.787380:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763195.787390:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496211072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763195.787397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763195.787400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763195.787401:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763195.787403:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3175 00000100:00080000:2.0:1789763195.787405:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763195.787410:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763195.787414:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763195.787421:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496211200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763195.787428:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763195.787431:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763195.787432:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763195.787435:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3175 00000100:00080000:2.0:1789763195.787439:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763195.787442:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763195.787445:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763195.787453:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496210944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763195.787459:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496211328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763195.787546:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763195.787605:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763195/real 1789763195] req@ffff8dd205267800 x1876699496210944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763195.787628:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496210944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763195.787641:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763195.787644:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763195.787655:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496210944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763195.787663:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496211072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763195.787683:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763195.787693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763195/real 1789763195] req@ffff8dd205264380 x1876699496211072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763195.787702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496211072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763195.787708:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763195.787710:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763195.787713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496211072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763195.787716:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496211200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763195.787728:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763195.787734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763195/real 1789763195] req@ffff8dd205264e00 x1876699496211200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763195.787741:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496211200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763195.787748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763195.787749:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763195.787751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496211200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763195.787777:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496211328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763195.787788:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763195.787794:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763195/real 1789763195] req@ffff8dd205267100 x1876699496211328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763195.787802:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496211328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763195.787808:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763195.787810:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763195.787812:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496211328:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763196.363343:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763197.387374:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763198.411368:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763199.435405:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763200.459393:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763200.907313:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763200.907326:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763200.907332:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763200.907346:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3180 00000100:00080000:2.0:1789763200.907360:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763200.907368:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763200.907417:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496211456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763200.907452:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763200.907457:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763200.907460:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763200.907465:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3180 00000100:00080000:2.0:1789763200.907469:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763200.907473:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763200.907485:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763200.907501:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496211584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763200.907516:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763200.907522:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763200.907525:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763200.907529:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3180 00000100:00080000:2.0:1789763200.907534:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763200.907543:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763200.907547:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496211456:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789763200.907552:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763200.907569:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496211712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763200.907583:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763200.907587:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763200.907589:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763200.907594:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3180 00000100:00080000:2.0:1789763200.907600:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763200.907605:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763200.907611:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763200.907626:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496211840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763200.907642:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763200.907701:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763200/real 1789763200] req@ffff8dd205267b80 x1876699496211456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763200.907726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496211456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763200.907740:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763200.907745:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763200.907751:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496211456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763200.907760:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496211584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763200.907786:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763200.907800:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763200/real 1789763200] req@ffff8dd205264000 x1876699496211584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763200.907816:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496211584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763200.907835:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763200.907839:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763200.907843:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496211584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763200.907881:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496211712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763200.907920:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763200.907944:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763200/real 1789763200] req@ffff8dd205265880 x1876699496211712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763200.907969:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496211712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763200.907981:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763200.907984:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763200.907988:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496211712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763200.907996:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496211840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763200.908050:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763200.908065:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763200/real 1789763200] req@ffff8dd205266a00 x1876699496211840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763200.908085:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496211840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763200.908097:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763200.908103:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763200.908109:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496211840:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763201.483331:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763202.507512:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763203.531454:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763204.555421:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763205.579427:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763206.027310:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763206.027328:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763206.027336:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763206.027353:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3185 00000100:00080000:2.0:1789763206.027372:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763206.027385:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763206.027444:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496211968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763206.027498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763206.027503:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763206.027506:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763206.027512:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3185 00000100:00080000:2.0:1789763206.027518:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763206.027524:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763206.027537:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763206.027558:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496212096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763206.027573:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763206.027578:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763206.027581:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763206.027602:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496211968:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763206.027603:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3185 00000100:00080000:2.0:1789763206.027613:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763206.027624:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763206.027631:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763206.027646:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496212224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763206.027659:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763206.027663:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763206.027666:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763206.027671:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3185 00000100:00080000:2.0:1789763206.027678:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763206.027686:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763206.027696:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763206.027719:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496212352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763206.027748:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763206.027828:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763206/real 1789763206] req@ffff8dd205264700 x1876699496211968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763206.027861:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496211968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763206.027882:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763206.027889:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763206.027897:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496211968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763206.027933:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496212096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763206.027978:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763206.028002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763206/real 1789763206] req@ffff8dd205265f80 x1876699496212096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763206.028025:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496212096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763206.028041:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763206.028047:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763206.028053:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496212096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763206.028064:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496212224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763206.028100:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763206.028122:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763206/real 1789763206] req@ffff8dd205265180 x1876699496212224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763206.028144:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496212224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763206.028159:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763206.028165:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763206.028270:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496212224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763206.028283:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496212352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763206.028331:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763206.028354:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763206/real 1789763206] req@ffff8dd205266d80 x1876699496212352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763206.028377:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496212352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763206.028392:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763206.028397:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763206.028403:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496212352:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763206.603414:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763207.627456:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763208.651450:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763209.675383:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763210.699396:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763211.147384:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763211.147397:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763211.147404:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763211.147417:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3190 00000100:00080000:3.0:1789763211.147432:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763211.147443:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763211.147493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496212480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763211.147526:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763211.147531:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763211.147534:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763211.147539:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3190 00000100:00080000:3.0:1789763211.147544:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763211.147550:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763211.147562:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763211.147581:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496212480:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763211.147582:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496212608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763211.147608:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763211.147612:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763211.147614:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763211.147619:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3190 00000100:00080000:3.0:1789763211.147624:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763211.147629:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763211.147635:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763211.147651:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496212736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763211.147664:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763211.147668:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763211.147670:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763211.147675:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3190 00000100:00080000:3.0:1789763211.147681:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763211.147686:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763211.147695:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763211.147717:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496212864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763211.147766:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496212608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763211.147833:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496212736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763211.147859:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496212864:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763211.723400:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763212.747410:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763213.771457:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763214.219751:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789763214.219821:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763214.219848:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763214.219916:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763214.219935:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763214.219957:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1789763214.220002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763211/real 1789763214] req@ffff8dd203145180 x1876699496212608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763214.220043:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496212608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763214.220065:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763214.220076:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763214.220085:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496212608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763214.220100:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763211/real 1789763214] req@ffff8dd203147800 x1876699496212736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763214.220125:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496212736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763214.220240:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763214.220269:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763214.220278:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496212736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763214.220292:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763211/real 1789763214] req@ffff8dd203146300 x1876699496212864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763214.220321:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496212864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763214.220339:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763214.220347:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763214.220353:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496212864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763214.220367:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763211/real 1789763214] req@ffff8dd203145c00 x1876699496212480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763214.220388:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496212480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763214.220402:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763214.220408:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763214.220414:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496212480:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789763214.795373:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763215.819373:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763216.267282:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763216.267296:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763216.267302:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763216.267317:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3195 00000100:00080000:0.0:1789763216.267333:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763216.267345:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763216.267400:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496212992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763216.267449:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763216.267458:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763216.267461:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763216.267470:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3195 00000100:00080000:0.0:1789763216.267478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763216.267487:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763216.267505:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763216.267539:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496213120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763216.267582:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496212992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763216.267583:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763216.267597:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763216.267602:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763216.267630:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3195 00000100:00080000:0.0:1789763216.267641:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763216.267650:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763216.267663:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763216.267699:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496213248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763216.267722:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763216.267728:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763216.267743:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763216.267751:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3195 00000100:00080000:0.0:1789763216.267762:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763216.267772:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763216.267783:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763216.267786:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763216.267821:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496213376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763216.267896:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763216/real 1789763216] req@ffff8dd211836d80 x1876699496212992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763216.267941:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496212992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763216.267967:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763216.267976:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763216.267984:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496212992:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763216.268040:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496213120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763216.268112:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763216.268280:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763216/real 1789763216] req@ffff8dd211835180 x1876699496213120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763216.268314:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496213120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763216.268331:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763216.268338:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763216.268345:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496213120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763216.268360:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496213248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763216.268418:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763216.268460:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763216/real 1789763216] req@ffff8dd211837480 x1876699496213248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763216.268488:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496213248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763216.268521:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763216.268530:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763216.268539:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496213248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763216.268554:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496213376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763216.268600:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763216.268637:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763216/real 1789763216] req@ffff8dd211834e00 x1876699496213376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763216.268665:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496213376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763216.268683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763216.268690:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763216.268696:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496213376:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763216.843382:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763217.867364:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763218.891430:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763219.915392:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763220.939422:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763221.387269:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763221.387286:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763221.387293:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763221.387339:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3200 00000100:00080000:0.0:1789763221.387353:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763221.387363:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763221.387413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496213504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763221.387451:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763221.387459:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763221.387463:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763221.387471:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3200 00000100:00080000:0.0:1789763221.387478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763221.387486:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763221.387501:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763221.387526:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496213632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763221.387527:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496213504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763221.387547:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763221.387553:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763221.387568:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763221.387575:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3200 00000100:00080000:0.0:1789763221.387581:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763221.387589:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763221.387598:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763221.387618:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496213760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763221.387636:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763221.387642:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763221.387645:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763221.387652:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3200 00000800:00000100:2.0:1789763221.387655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763221.387657:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763221.387663:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763221.387672:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763221.387689:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496213888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763221.387713:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763221/real 1789763221] req@ffff8dd211834e00 x1876699496213504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763221.387745:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496213504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763221.387780:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763221.387788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763221.387796:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496213504:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763221.387832:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496213632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763221.387870:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763221.387895:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763221/real 1789763221] req@ffff8dd211837480 x1876699496213632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763221.387911:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496213632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763221.387922:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763221.387926:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763221.387931:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496213632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763221.387938:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496213760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763221.387960:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763221.387971:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763221/real 1789763221] req@ffff8dd211835180 x1876699496213760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763221.387986:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496213760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763221.387996:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763221.387999:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763221.388003:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496213760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763221.388009:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496213888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763221.388028:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763221.388039:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763221/real 1789763221] req@ffff8dd211836d80 x1876699496213888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763221.388054:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496213888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763221.388064:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763221.388067:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763221.388071:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496213888:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763221.963431:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763222.987373:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763224.011328:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763225.035400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763226.059395:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763226.507318:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763226.507331:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763226.507336:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763226.507349:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3206 00000100:00080000:3.0:1789763226.507364:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763226.507376:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763226.507433:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496214016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763226.507482:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763226.507487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763226.507500:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763226.507507:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3206 00000100:00080000:3.0:1789763226.507514:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763226.507521:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763226.507535:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763226.507560:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496214016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763226.507561:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496214144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763226.507594:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763226.507602:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763226.507607:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763226.507616:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3206 00000100:00080000:3.0:1789763226.507639:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763226.507651:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1789763226.507655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1789763226.507682:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763226.507715:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496214272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763226.507739:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763226/real 1789763226] req@ffff8dd203144700 x1876699496214016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763226.507742:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763226.507759:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763226.507767:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763226.507781:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3206 00000100:00080000:3.0:1789763226.507789:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763226.507803:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496214016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763226.507818:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763226.507830:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763226.507851:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763226.507861:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763226.507872:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496214016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763226.507874:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496214400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763226.507913:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496214144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763226.507955:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763226.507974:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763226/real 1789763226] req@ffff8dd203145f80 x1876699496214144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763226.507991:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496214144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763226.508010:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763226.508014:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763226.508019:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496214144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763226.508026:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496214272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763226.508047:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763226.508086:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763226/real 1789763226] req@ffff8dd203144a80 x1876699496214272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763226.508105:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496214272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763226.508116:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763226.508120:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763226.508124:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496214272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763226.508132:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496214400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763226.508159:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763226.508213:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763226/real 1789763226] req@ffff8dd203145500 x1876699496214400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763226.508231:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496214400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763226.508241:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763226.508245:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763226.508250:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496214400:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763227.083391:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763228.107389:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763229.131404:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763230.155363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763231.179394:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763231.627330:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763231.627359:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763231.627366:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763231.627383:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3211 00000100:00080000:1.0:1789763231.627398:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763231.627407:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763231.627455:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496214528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763231.627493:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763231.627498:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763231.627501:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763231.627506:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3211 00000100:00080000:1.0:1789763231.627510:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763231.627514:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763231.627526:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763231.627570:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496214528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763231.627572:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496214656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763231.627604:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763231.627611:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763231.627616:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763231.627625:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3211 00000100:00080000:1.0:1789763231.627634:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763231.627643:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763231.627653:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789763231.627661:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763231.627680:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496214784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763231.627696:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763231.627701:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763231.627704:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763231.627712:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3211 00000100:00080000:1.0:1789763231.627721:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763231.627731:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763231/real 1789763231] req@ffff8dd20504c000 x1876699496214528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763231.627734:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763231.627750:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763231.627791:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496214912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763231.627803:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496214528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763231.627832:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763231.627839:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763231.627844:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496214528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763231.627879:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496214656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763231.627916:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763231.627934:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763231/real 1789763231] req@ffff8dd20504ca80 x1876699496214656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763231.627950:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496214656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763231.627961:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763231.627966:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763231.627970:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496214656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763231.627978:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496214784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763231.627998:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763231.628012:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763231/real 1789763231] req@ffff8dd20504ea00 x1876699496214784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763231.628071:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496214784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763231.628083:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763231.628087:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763231.628091:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496214784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763231.628099:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496214912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763231.628127:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763231.628141:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763231/real 1789763231] req@ffff8dd20504c380 x1876699496214912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763231.628156:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496214912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763231.628165:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763231.628218:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763231.628222:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496214912:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763232.203341:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763233.227344:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763234.251329:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763235.275379:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763236.299368:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763236.747326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763236.747339:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763236.747345:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763236.747359:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3216 00000100:00080000:2.0:1789763236.747374:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763236.747383:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763236.747445:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496215040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763236.747485:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763236.747490:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763236.747493:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763236.747498:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3216 00000100:00080000:2.0:1789763236.747503:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763236.747507:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763236.747520:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763236.747537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496215168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763236.747550:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763236.747554:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763236.747556:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763236.747560:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3216 00000100:00080000:2.0:1789763236.747564:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763236.747569:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763236.747575:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763236.747587:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496215296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763236.747601:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763236.747606:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763236.747608:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763236.747617:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496215040:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763236.747620:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3216 00000100:00080000:2.0:1789763236.747630:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763236.747636:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763236.747642:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763236.747657:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496215424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763236.747713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763236.747767:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763236/real 1789763236] req@ffff8dd205266300 x1876699496215040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763236.747791:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496215040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763236.747814:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763236.747819:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763236.747824:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496215040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763236.747833:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496215168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763236.747857:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763236.747871:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763236/real 1789763236] req@ffff8dd205266d80 x1876699496215168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763236.747886:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496215168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763236.747896:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763236.747900:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763236.747904:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496215168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763236.747932:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496215296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763236.747953:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763236.747964:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763236/real 1789763236] req@ffff8dd205265180 x1876699496215296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763236.747978:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496215296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763236.748015:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763236.748019:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763236.748023:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496215296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763236.748031:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496215424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763236.748054:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763236.748067:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763236/real 1789763236] req@ffff8dd205265f80 x1876699496215424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763236.748081:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496215424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763236.748091:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763236.748095:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763236.748098:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496215424:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763237.323351:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763238.347379:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763239.371464:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763240.395366:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763241.419375:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763241.867290:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763241.867303:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763241.867308:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763241.867322:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3221 00000100:00080000:2.0:1789763241.867335:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763241.867345:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763241.867394:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496215552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763241.867431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763241.867436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763241.867438:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763241.867444:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3221 00000100:00080000:2.0:1789763241.867448:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763241.867452:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763241.867464:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763241.867481:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496215680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763241.867497:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763241.867503:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763241.867507:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763241.867513:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496215552:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763241.867517:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3221 00000100:00080000:2.0:1789763241.867528:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763241.867534:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763241.867541:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763241.867571:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496215808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763241.867586:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763241.867590:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763241.867593:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763241.867598:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3221 00000100:00080000:2.0:1789763241.867605:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763241.867610:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763241.867616:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763241.867628:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763241.867633:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496215936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763241.867711:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763241/real 1789763241] req@ffff8dd205264700 x1876699496215552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763241.867750:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496215552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763241.867772:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763241.867781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763241.867787:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496215552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763241.867818:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496215680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763241.867856:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763241.867874:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763241/real 1789763241] req@ffff8dd205266a00 x1876699496215680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763241.867892:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496215680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763241.867903:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763241.867908:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763241.867912:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496215680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763241.867919:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496215808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763241.867940:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763241.867982:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763241/real 1789763241] req@ffff8dd205265880 x1876699496215808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763241.868008:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496215808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763241.868020:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763241.868024:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763241.868028:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496215808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763241.868036:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496215936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763241.868063:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763241.868076:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763241/real 1789763241] req@ffff8dd205264000 x1876699496215936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763241.868109:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496215936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763241.868129:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763241.868133:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763241.868137:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496215936:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763242.443361:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763243.467313:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763244.491393:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763245.515358:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763246.539367:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763246.987288:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763246.987301:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763246.987307:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763246.987320:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3226 00000100:00080000:2.0:1789763246.987336:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763246.987348:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763246.987396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496216064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763246.987428:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763246.987433:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763246.987436:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763246.987441:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3226 00000100:00080000:2.0:1789763246.987445:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763246.987449:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763246.987460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763246.987477:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496216192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763246.987491:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763246.987496:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763246.987498:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763246.987503:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3226 00000100:00080000:2.0:1789763246.987507:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763246.987517:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496216064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763246.987518:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763246.987532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763246.987551:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496216320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763246.987566:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763246.987572:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763246.987576:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763246.987584:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3226 00000100:00080000:2.0:1789763246.987594:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763246.987603:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1789763246.987613:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789763246.987614:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763246.987645:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496216448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763246.987703:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763246/real 1789763246] req@ffff8dd205267b80 x1876699496216064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763246.987743:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496216064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763246.987759:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763246.987764:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763246.987770:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496216064:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763246.987780:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496216192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763246.987816:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763246.987833:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763246/real 1789763246] req@ffff8dd205267100 x1876699496216192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763246.987849:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496216192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763246.987860:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763246.987864:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763246.987869:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496216192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763246.987899:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496216320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763246.987957:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763246.987973:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763246/real 1789763246] req@ffff8dd205264e00 x1876699496216320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763246.987989:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496216320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763246.988000:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763246.988004:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763246.988009:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496216320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763246.988016:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496216448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763246.988037:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763246.988048:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763246/real 1789763246] req@ffff8dd205264380 x1876699496216448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763246.988062:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496216448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763246.988073:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763246.988076:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763246.988080:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496216448:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763247.563399:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763248.587372:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763249.611384:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763250.635392:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763251.659425:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763252.107365:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763252.107382:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763252.107388:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763252.107405:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3231 00000100:00080000:0.0:1789763252.107424:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763252.107436:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763252.107500:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496216576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763252.107544:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763252.107550:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763252.107553:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763252.107559:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3231 00000100:00080000:0.0:1789763252.107563:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763252.107567:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763252.107585:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763252.107600:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496216704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763252.107614:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763252.107618:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763252.107622:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763252.107627:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3231 00000100:00080000:0.0:1789763252.107631:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763252.107636:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763252.107643:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763252.107656:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496216832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763252.107670:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763252.107678:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1789763252.107683:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496216576:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763252.107684:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763252.107701:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3231 00000100:00080000:0.0:1789763252.107708:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763252.107715:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763252.107726:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763252.107751:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496216960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763252.107806:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763252.107899:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763252/real 1789763252] req@ffff8dd211834a80 x1876699496216576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763252.107929:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496216576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763252.107945:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763252.107951:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763252.107957:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496216576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763252.107968:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496216704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763252.107997:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763252.108013:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763252/real 1789763252] req@ffff8dd211836680 x1876699496216704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763252.108029:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496216704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763252.108040:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763252.108044:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763252.108048:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496216704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763252.108083:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496216832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763252.108106:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763252.108118:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763252/real 1789763252] req@ffff8dd211834380 x1876699496216832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763252.108133:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496216832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763252.108152:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763252.108157:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763252.108161:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496216832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763252.108217:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496216960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763252.108243:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763252.108257:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763252/real 1789763252] req@ffff8dd211835f80 x1876699496216960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763252.108272:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496216960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763252.108282:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763252.108286:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763252.108290:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496216960:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763252.683374:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763253.707402:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763254.731450:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763255.755441:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763256.779435:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763257.227340:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763257.227354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763257.227360:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763257.227374:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3236 00000100:00080000:2.0:1789763257.227388:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763257.227397:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763257.227446:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496217088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763257.227485:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763257.227491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763257.227493:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763257.227499:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3236 00000100:00080000:2.0:1789763257.227503:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763257.227507:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763257.227519:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763257.227534:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496217216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763257.227549:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763257.227554:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763257.227557:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763257.227563:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3236 00000100:00080000:2.0:1789763257.227569:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763257.227577:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763257.227588:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763257.227596:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496217088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763257.227616:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496217344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763257.227643:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763257.227651:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763257.227656:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763257.227680:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3236 00000800:00000100:3.0:1789763257.227690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763257.227695:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763257.227706:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763257.227716:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763257.227739:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496217472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763257.227758:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763257/real 1789763257] req@ffff8dd205267800 x1876699496217088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763257.227787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496217088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763257.227803:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763257.227809:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763257.227814:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496217088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763257.227824:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496217216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763257.227884:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763257.227903:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763257/real 1789763257] req@ffff8dd205264a80 x1876699496217216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763257.227919:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496217216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763257.227930:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763257.227935:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763257.227939:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496217216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763257.227970:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496217344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763257.227993:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763257.228005:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763257/real 1789763257] req@ffff8dd205267480 x1876699496217344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763257.228020:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496217344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763257.228030:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763257.228033:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763257.228037:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496217344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763257.228059:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496217472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763257.228081:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763257.228093:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763257/real 1789763257] req@ffff8dd205266680 x1876699496217472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763257.228108:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496217472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763257.228118:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763257.228122:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763257.228126:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496217472:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763257.803378:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763258.827484:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763259.851311:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763260.875416:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763261.899401:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763262.347290:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763262.347303:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763262.347308:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763262.347320:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3241 00000100:00080000:2.0:1789763262.347334:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763262.347343:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763262.347391:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496217600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763262.347429:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763262.347435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763262.347438:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763262.347443:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3241 00000100:00080000:2.0:1789763262.347447:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763262.347451:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763262.347462:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763262.347478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496217728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763262.347493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763262.347498:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763262.347501:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763262.347505:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3241 00000100:00080000:2.0:1789763262.347509:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763262.347519:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763262.347521:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496217600:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789763262.347526:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763262.347556:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496217856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763262.347577:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763262.347585:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763262.347589:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763262.347597:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3241 00000100:00080000:2.0:1789763262.347608:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763262.347618:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763262.347629:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763262.347633:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763262.347661:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496217984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763262.347703:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763262/real 1789763262] req@ffff8dd205265500 x1876699496217600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763262.347730:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496217600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763262.347744:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763262.347750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763262.347755:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496217600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763262.347765:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496217728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763262.347819:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763262.347839:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763262/real 1789763262] req@ffff8dd205266680 x1876699496217728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763262.347855:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496217728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763262.347875:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763262.347880:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763262.347884:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496217728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763262.347913:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496217856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763262.347940:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763262.347953:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763262/real 1789763262] req@ffff8dd205267480 x1876699496217856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763262.347967:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496217856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763262.347977:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763262.347981:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763262.347985:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496217856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763262.347992:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496217984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763262.348010:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763262.348021:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763262/real 1789763262] req@ffff8dd205264a80 x1876699496217984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763262.348035:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496217984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763262.348045:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763262.348048:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763262.348053:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496217984:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763262.923447:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763263.947395:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763264.971419:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763265.995407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763267.019415:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763267.467320:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763267.467338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763267.467347:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763267.467364:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3247 00000100:00080000:1.0:1789763267.467382:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763267.467392:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763267.467439:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496218112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763267.467477:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763267.467483:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763267.467486:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763267.467492:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3247 00000100:00080000:1.0:1789763267.467496:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763267.467501:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763267.467514:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763267.467544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496218240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763267.467561:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763267.467569:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789763267.467569:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496218112:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763267.467576:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763267.467592:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3247 00000100:00080000:1.0:1789763267.467596:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763267.467600:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763267.467607:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763267.467624:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496218368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763267.467638:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763267.467642:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763267.467645:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763267.467650:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3247 00000100:00080000:1.0:1789763267.467655:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763267.467658:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763267.467664:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763267.467680:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496218496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763267.467695:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763267.467793:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763267/real 1789763267] req@ffff8dd20504dc00 x1876699496218112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763267.467820:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496218112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763267.467834:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763267.467840:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763267.467845:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496218112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763267.467877:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496218240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763267.467912:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763267.467928:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763267/real 1789763267] req@ffff8dd20504c380 x1876699496218240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763267.467944:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496218240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763267.467955:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763267.467959:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763267.467963:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496218240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763267.467971:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496218368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763267.467992:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763267.468004:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763267/real 1789763267] req@ffff8dd20504ea00 x1876699496218368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763267.468018:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496218368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763267.468028:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763267.468032:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763267.468035:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496218368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763267.468042:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496218496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763267.468061:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763267.468073:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763267/real 1789763267] req@ffff8dd20504ca80 x1876699496218496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763267.468087:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496218496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763267.468096:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763267.468100:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763267.468103:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496218496:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763268.043393:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763269.067402:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763270.091452:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763271.115384:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763272.139415:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763272.587347:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763272.587364:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763272.587371:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763272.587387:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3252 00000100:00080000:2.0:1789763272.587404:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763272.587417:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763272.587468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496218624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763272.587506:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763272.587511:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763272.587514:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763272.587519:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3252 00000100:00080000:2.0:1789763272.587524:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763272.587529:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763272.587541:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763272.587559:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496218752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763272.587576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763272.587578:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496218624:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763272.587587:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763272.587595:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763272.587601:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3252 00000100:00080000:2.0:1789763272.587604:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763272.587609:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763272.587617:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763272.587638:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496218880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763272.587659:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763272.587666:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763272.587671:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763272.587679:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763272.587688:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3252 00000100:00080000:2.0:1789763272.587701:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763272.587710:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763272.587744:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763272.587774:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496219008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763272.587783:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763272/real 1789763272] req@ffff8dd205267800 x1876699496218624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763272.587836:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496218624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763272.587853:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763272.587859:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763272.587865:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496218624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763272.587895:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496218752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763272.587928:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763272.587944:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763272/real 1789763272] req@ffff8dd205264380 x1876699496218752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763272.587960:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496218752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763272.587971:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763272.587976:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763272.587980:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496218752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763272.587987:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496218880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763272.588007:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763272.588018:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763272/real 1789763272] req@ffff8dd205264e00 x1876699496218880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763272.588033:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496218880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763272.588043:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763272.588047:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763272.588051:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496218880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763272.588058:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496219008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763272.588077:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763272.588088:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763272/real 1789763272] req@ffff8dd205267100 x1876699496219008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763272.588103:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496219008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763272.588112:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763272.588116:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763272.588119:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496219008:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763273.163409:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763274.187418:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763275.211447:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763276.235431:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763277.259409:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763277.707226:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763277.707237:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763277.707242:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763277.707252:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3257 00000100:00080000:2.0:1789763277.707260:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763277.707266:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763277.707298:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496219136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763277.707322:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763277.707326:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763277.707328:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763277.707332:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3257 00000100:00080000:2.0:1789763277.707336:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763277.707340:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763277.707350:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763277.707362:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496219136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763277.707368:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496219264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763277.707381:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763277.707383:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763277.707385:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763277.707388:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3257 00000100:00080000:2.0:1789763277.707391:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763277.707394:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763277.707399:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763277.707411:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496219392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763277.707422:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763277.707426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763277.707428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763277.707433:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3257 00000100:00080000:2.0:1789763277.707437:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763277.707442:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763277.707448:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763277.707459:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496219520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763277.707466:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496219264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763277.707500:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496219392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763277.707517:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496219520:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763278.283363:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763279.307408:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763280.331427:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763280.779504:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789763280.779557:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763280.779593:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763280.779652:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763280.779697:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763280.779719:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789763280.779746:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763277/real 1789763280] req@ffff8dd205264000 x1876699496219264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.779783:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496219264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.779799:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.779805:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763280.779811:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496219264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763280.779821:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763277/real 1789763280] req@ffff8dd205265880 x1876699496219392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.779836:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496219392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.779846:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.779850:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763280.779854:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496219392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763280.779862:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763277/real 1789763280] req@ffff8dd205267b80 x1876699496219136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.779876:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496219136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.779886:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.779906:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763280.779913:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496219136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763280.779952:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763277/real 1789763280] req@ffff8dd205266a00 x1876699496219520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763280.779989:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496219520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763280.780008:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763280.780015:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763280.780023:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496219520:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763281.355299:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763282.379311:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763282.827242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763282.827256:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763282.827260:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763282.827279:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3262 00000100:00080000:2.0:1789763282.827293:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763282.827302:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763282.827352:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496219648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763282.827385:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763282.827390:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763282.827392:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763282.827398:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3262 00000100:00080000:2.0:1789763282.827403:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763282.827407:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763282.827419:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763282.827427:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496219648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763282.827435:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496219776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763282.827445:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763282.827449:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763282.827452:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763282.827457:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3262 00000100:00080000:2.0:1789763282.827461:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763282.827466:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763282.827472:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763282.827486:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496219904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763282.827498:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763282.827502:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763282.827504:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763282.827509:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3262 00000100:00080000:2.0:1789763282.827513:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763282.827520:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1789763282.827521:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789763282.827528:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763282.827544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496220032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763282.827606:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763282/real 1789763282] req@ffff8dd205264700 x1876699496219648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763282.827633:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496219648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763282.827683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763282.827689:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763282.827694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496219648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763282.827724:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496219776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763282.827748:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763282.827759:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763282/real 1789763282] req@ffff8dd205265f80 x1876699496219776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763282.827768:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496219776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763282.827775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763282.827778:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763282.827781:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496219776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763282.827790:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496219904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763282.827805:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763282.827812:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763282/real 1789763282] req@ffff8dd205265180 x1876699496219904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763282.827821:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496219904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763282.827828:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763282.827830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763282.827832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496219904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763282.827836:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496220032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763282.827847:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763282.827852:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763282/real 1789763282] req@ffff8dd205266d80 x1876699496220032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763282.827860:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496220032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763282.827865:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763282.827867:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763282.827869:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496220032:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763283.403455:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763284.427305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763285.451309:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763286.475292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763287.499402:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763287.947301:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763287.947312:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763287.947316:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763287.947327:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3267 00000100:00080000:1.0:1789763287.947338:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763287.947345:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763287.947387:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496220160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763287.947423:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763287.947426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763287.947428:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763287.947432:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3267 00000100:00080000:1.0:1789763287.947436:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763287.947440:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763287.947451:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763287.947463:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496220160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763287.947464:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496220288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763287.947488:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763287.947491:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763287.947492:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763287.947495:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3267 00000100:00080000:1.0:1789763287.947496:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763287.947499:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763287.947503:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763287.947512:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496220416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763287.947519:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763287.947522:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763287.947523:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763287.947526:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3267 00000100:00080000:1.0:1789763287.947529:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763287.947531:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763287.947534:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763287.947541:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496220544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763287.947567:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763287.947640:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763287/real 1789763287] req@ffff8dd20504c000 x1876699496220160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763287.947658:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496220160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763287.947683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763287.947686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763287.947690:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496220160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763287.947713:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496220288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763287.947732:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763287.947742:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763287/real 1789763287] req@ffff8dd20504d880 x1876699496220288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763287.947750:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496220288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763287.947756:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763287.947759:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763287.947761:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496220288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763287.947765:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496220416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763287.947777:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763287.947784:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763287/real 1789763287] req@ffff8dd20504ed80 x1876699496220416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763287.947791:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496220416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763287.947797:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763287.947799:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763287.947801:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496220416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763287.947805:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496220544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763287.947817:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763287.947823:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763287/real 1789763287] req@ffff8dd20504f100 x1876699496220544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763287.947830:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496220544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763287.947836:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763287.947837:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763287.947839:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496220544:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763288.523368:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763289.547312:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763290.571301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763291.595290:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763292.619302:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763293.067286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763293.067295:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763293.067298:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763293.067307:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3272 00000100:00080000:0.0:1789763293.067317:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763293.067324:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763293.067359:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496220672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763293.067385:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763293.067388:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763293.067390:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763293.067393:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3272 00000100:00080000:0.0:1789763293.067396:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763293.067398:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763293.067406:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763293.067415:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496220800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763293.067423:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763293.067426:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763293.067428:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763293.067431:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3272 00000100:00080000:0.0:1789763293.067434:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763293.067437:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763293.067440:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763293.067447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496220928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763293.067456:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763293.067460:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1789763293.067462:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496220672:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763293.067463:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763293.067471:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3272 00000100:00080000:0.0:1789763293.067474:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763293.067477:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763293.067480:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763293.067490:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496221056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763293.067531:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763293.067603:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763293/real 1789763293] req@ffff8dd211835c00 x1876699496220672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763293.067619:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496220672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763293.067627:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763293.067647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763293.067651:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496220672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763293.067659:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496220800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763293.067682:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763293.067694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763293/real 1789763293] req@ffff8dd211834700 x1876699496220800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763293.067705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496220800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763293.067713:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763293.067716:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763293.067719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496220800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763293.067741:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496220928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763293.067757:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763293.067765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763293/real 1789763293] req@ffff8dd211834000 x1876699496220928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763293.067776:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496220928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763293.067783:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763293.067786:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763293.067789:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496220928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763293.067793:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496221056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763293.067806:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763293.067814:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763293/real 1789763293] req@ffff8dd211836a00 x1876699496221056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763293.067824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496221056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763293.067837:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763293.067840:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763293.067842:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496221056:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763293.643286:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763294.667271:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763295.691284:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763296.715323:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763297.739302:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763298.187260:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763298.187271:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763298.187276:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763298.187286:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3277 00000100:00080000:1.0:1789763298.187297:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763298.187305:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763298.187340:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496221184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763298.187361:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763298.187364:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763298.187366:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763298.187369:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3277 00000100:00080000:1.0:1789763298.187371:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763298.187374:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763298.187381:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763298.187391:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496221312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763298.187398:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763298.187402:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763298.187405:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763298.187406:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496221184:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763298.187409:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3277 00000100:00080000:1.0:1789763298.187415:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763298.187419:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763298.187422:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763298.187431:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496221440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763298.187438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763298.187441:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763298.187443:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763298.187445:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3277 00000100:00080000:1.0:1789763298.187448:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763298.187451:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763298.187453:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763298.187460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496221568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763298.187475:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763298.187513:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763298/real 1789763298] req@ffff8dd20504c700 x1876699496221184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763298.187546:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496221184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763298.187558:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763298.187562:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763298.187566:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496221184:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763298.187590:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496221312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763298.187612:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763298.187623:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763298/real 1789763298] req@ffff8dd20504f800 x1876699496221312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763298.187637:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496221312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763298.187646:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763298.187649:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763298.187653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496221312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763298.187660:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496221440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763298.187676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763298.187686:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763298/real 1789763298] req@ffff8dd20504fb80 x1876699496221440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763298.187698:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496221440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763298.187706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763298.187710:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763298.187713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496221440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763298.187718:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496221568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763298.187734:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763298.187744:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763298/real 1789763298] req@ffff8dd20504ce00 x1876699496221568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763298.187755:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496221568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763298.187764:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763298.187767:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763298.187770:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496221568:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763298.763299:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763299.787299:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763300.811291:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763301.835310:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763302.859292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763303.307274:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763303.307282:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763303.307285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763303.307293:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3282 00000100:00080000:2.0:1789763303.307302:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763303.307308:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763303.307339:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496221696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763303.307361:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763303.307363:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763303.307365:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763303.307367:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3282 00000100:00080000:2.0:1789763303.307370:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763303.307372:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763303.307379:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763303.307390:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496221824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763303.307398:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496221696:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763303.307399:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763303.307406:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763303.307409:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763303.307412:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3282 00000100:00080000:2.0:1789763303.307414:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763303.307416:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763303.307419:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763303.307427:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496221952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763303.307433:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763303.307435:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763303.307436:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763303.307439:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3282 00000100:00080000:2.0:1789763303.307442:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763303.307444:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763303.307447:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763303.307453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496222080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763303.307471:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763303.307518:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763303/real 1789763303] req@ffff8dd205266300 x1876699496221696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763303.307533:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496221696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763303.307541:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763303.307544:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763303.307547:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496221696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763303.307565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496221824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763303.307583:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763303.307592:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763303/real 1789763303] req@ffff8dd205266d80 x1876699496221824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763303.307600:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496221824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763303.307606:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763303.307608:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763303.307610:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496221824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763303.307614:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496221952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763303.307625:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763303.307630:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763303/real 1789763303] req@ffff8dd205265180 x1876699496221952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763303.307638:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496221952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763303.307647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763303.307650:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763303.307652:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496221952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763303.307655:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496222080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763303.307670:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763303.307680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763303/real 1789763303] req@ffff8dd205265f80 x1876699496222080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763303.307692:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496222080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763303.307701:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763303.307705:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763303.307708:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496222080:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763303.883322:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763304.907331:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763305.931315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763306.955315:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763307.979303:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763308.427235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763308.427245:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763308.427249:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763308.427259:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3288 00000100:00080000:2.0:1789763308.427269:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763308.427276:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763308.427310:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496222208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763308.427336:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763308.427339:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763308.427341:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763308.427345:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3288 00000100:00080000:2.0:1789763308.427349:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763308.427354:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763308.427364:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763308.427374:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496222208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763308.427379:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496222336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763308.427393:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763308.427397:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763308.427399:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763308.427403:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3288 00000100:00080000:2.0:1789763308.427407:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763308.427411:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763308.427416:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763308.427428:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496222464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763308.427439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763308.427442:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763308.427443:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763308.427449:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763308.427450:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3288 00000100:00080000:2.0:1789763308.427469:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763308.427473:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763308.427479:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763308.427491:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496222592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763308.427512:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763308/real 1789763308] req@ffff8dd205264700 x1876699496222208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763308.427527:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496222208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763308.427535:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763308.427538:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763308.427541:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496222208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763308.427560:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496222336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763308.427578:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763308.427587:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763308/real 1789763308] req@ffff8dd205266a00 x1876699496222336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763308.427596:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496222336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763308.427602:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763308.427604:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763308.427606:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496222336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763308.427610:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496222464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763308.427620:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763308.427626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763308/real 1789763308] req@ffff8dd205267b80 x1876699496222464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763308.427634:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496222464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763308.427639:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763308.427641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763308.427644:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496222464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763308.427650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496222592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763308.427669:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763308.427680:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763308/real 1789763308] req@ffff8dd205265880 x1876699496222592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763308.427695:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496222592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763308.427703:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763308.427707:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763308.427710:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496222592:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763309.003326:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763310.027306:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763311.051277:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763312.075357:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763313.099469:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763313.547271:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763313.547283:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763313.547289:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763313.547302:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3293 00000100:00080000:2.0:1789763313.547315:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763313.547324:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763313.547372:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496222720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763313.547409:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763313.547414:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763313.547417:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763313.547444:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3293 00000100:00080000:2.0:1789763313.547449:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763313.547460:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763313.547472:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763313.547488:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496222848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763313.547502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763313.547507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763313.547510:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763313.547515:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3293 00000100:00080000:2.0:1789763313.547519:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763313.547524:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763313.547532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763313.547539:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496222720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763313.547550:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496222976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763313.547567:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763313.547572:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763313.547574:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763313.547579:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3293 00000100:00080000:2.0:1789763313.547586:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763313.547591:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763313.547598:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763313.547614:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496223104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763313.547632:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763313.547692:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763313/real 1789763313] req@ffff8dd205264000 x1876699496222720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763313.547717:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496222720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763313.547731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763313.547736:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763313.547742:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496222720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763313.547751:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496222848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763313.547776:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763313.547789:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763313/real 1789763313] req@ffff8dd205267100 x1876699496222848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763313.547804:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496222848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763313.547823:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763313.547827:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763313.547832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496222848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763313.547861:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496222976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763313.547881:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763313.547894:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763313/real 1789763313] req@ffff8dd205264e00 x1876699496222976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763313.547908:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496222976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763313.547918:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763313.547921:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763313.547925:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496222976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763313.547932:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496223104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763313.547961:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763313.547981:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763313/real 1789763313] req@ffff8dd205264380 x1876699496223104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763313.548005:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496223104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763313.548021:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763313.548028:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763313.548036:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496223104:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763314.123346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763315.147488:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763316.171378:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763317.195527:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763318.219461:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763318.667270:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763318.667284:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763318.667290:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763318.667304:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3298 00000100:00080000:2.0:1789763318.667318:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763318.667327:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763318.667378:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496223232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763318.667436:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763318.667441:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763318.667444:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763318.667449:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3298 00000100:00080000:2.0:1789763318.667454:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763318.667458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763318.667470:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763318.667486:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496223360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763318.667501:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763318.667507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763318.667510:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763318.667515:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3298 00000100:00080000:2.0:1789763318.667520:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763318.667525:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763318.667534:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763318.667543:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496223232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763318.667553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496223488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763318.667570:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763318.667574:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763318.667577:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763318.667582:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3298 00000100:00080000:2.0:1789763318.667586:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763318.667591:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763318.667597:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763318.667611:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496223616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763318.667638:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763318.667693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763318/real 1789763318] req@ffff8dd205267800 x1876699496223232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763318.667718:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496223232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763318.667732:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763318.667738:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763318.667743:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496223232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763318.667752:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496223360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763318.667790:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763318.667810:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763318/real 1789763318] req@ffff8dd205264a80 x1876699496223360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763318.667829:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496223360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763318.667846:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763318.667853:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763318.667860:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496223360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763318.667900:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496223488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763318.667942:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763318.667967:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763318/real 1789763318] req@ffff8dd205267480 x1876699496223488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763318.667985:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496223488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763318.667996:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763318.668000:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763318.668004:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496223488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763318.668012:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496223616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763318.668038:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763318.668051:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763318/real 1789763318] req@ffff8dd205266680 x1876699496223616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763318.668066:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496223616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763318.668075:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763318.668079:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763318.668083:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496223616:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763319.243479:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763320.267446:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763321.291282:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763322.315317:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763323.339473:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763323.787376:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763323.787393:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763323.787401:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763323.787419:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3303 00000100:00080000:0.0:1789763323.787440:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763323.787452:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763323.787520:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496223744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763323.787565:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763323.787571:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763323.787575:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763323.787581:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3303 00000100:00080000:0.0:1789763323.787585:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763323.787590:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763323.787606:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763323.787624:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496223872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763323.787639:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763323.787644:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763323.787647:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763323.787657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496223744:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763323.787658:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3303 00000100:00080000:0.0:1789763323.787664:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763323.787675:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763323.787686:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763323.787701:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496224000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763323.787714:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763323.787718:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763323.787720:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763323.787738:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3303 00000100:00080000:0.0:1789763323.787743:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763323.787748:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763323.787755:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763323.787769:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496224128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763323.787800:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763323.787872:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763323/real 1789763323] req@ffff8dd211837100 x1876699496223744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763323.787899:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496223744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763323.787916:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763323.787921:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763323.787927:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496223744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763323.787964:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496223872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763323.787994:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763323.788009:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763323/real 1789763323] req@ffff8dd211836300 x1876699496223872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763323.788024:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496223872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763323.788035:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763323.788040:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763323.788044:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496223872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763323.788051:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496224000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763323.788071:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763323.788082:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763323/real 1789763323] req@ffff8dd211837800 x1876699496224000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763323.788097:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496224000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763323.788108:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763323.788112:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763323.788116:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496224000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763323.788123:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496224128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763323.788143:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763323.788154:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763323/real 1789763323] req@ffff8dd211837b80 x1876699496224128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763323.788212:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496224128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763323.788224:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763323.788229:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763323.788232:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496224128:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763324.363457:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763325.387416:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763326.411433:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763327.435523:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763328.459403:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763328.907286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763328.907297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763328.907302:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763328.907323:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3308 00000100:00080000:1.0:1789763328.907336:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763328.907343:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763328.907395:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496224256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763328.907432:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763328.907436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763328.907438:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763328.907442:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3308 00000100:00080000:1.0:1789763328.907446:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763328.907449:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763328.907460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763328.907473:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496224384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763328.907483:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763328.907487:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763328.907488:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763328.907498:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496224256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763328.907499:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3308 00000100:00080000:1.0:1789763328.907505:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763328.907519:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763328.907523:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763328.907534:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496224512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763328.907543:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763328.907546:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763328.907548:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763328.907552:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3308 00000100:00080000:1.0:1789763328.907555:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763328.907558:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763328.907562:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763328.907570:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496224640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763328.907580:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763328.907619:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763328/real 1789763328] req@ffff8dd20504d180 x1876699496224256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763328.907635:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496224256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763328.907644:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763328.907647:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763328.907650:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496224256:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763328.907670:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496224384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763328.907685:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763328.907693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763328/real 1789763328] req@ffff8dd20504e300 x1876699496224384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763328.907701:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496224384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763328.907707:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763328.907709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763328.907711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496224384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763328.907715:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496224512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763328.907726:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763328.907732:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763328/real 1789763328] req@ffff8dd20504d500 x1876699496224512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763328.907739:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496224512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763328.907744:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763328.907746:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763328.907748:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496224512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763328.907752:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496224640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763328.907762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763328.907768:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763328/real 1789763328] req@ffff8dd20504df80 x1876699496224640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763328.907775:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496224640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763328.907785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763328.907787:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763328.907789:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496224640:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763329.483297:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763330.507395:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763331.531421:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763332.555438:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763333.579462:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763334.027420:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763334.027434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763334.027441:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763334.027455:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3313 00000100:00080000:2.0:1789763334.027469:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763334.027479:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763334.027527:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496224768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763334.027576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763334.027581:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763334.027584:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763334.027589:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3313 00000100:00080000:2.0:1789763334.027594:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763334.027600:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763334.027613:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763334.027631:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496224896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763334.027648:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763334.027657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496224768:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763334.027658:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763334.027667:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763334.027683:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3313 00000100:00080000:2.0:1789763334.027690:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763334.027699:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763334.027712:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763334.027740:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496225024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763334.027760:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763334.027792:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763334.027797:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763334.027800:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763334.027809:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3313 00000100:00080000:2.0:1789763334.027816:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763334.027823:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763334/real 1789763334] req@ffff8dd205265500 x1876699496224768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763334.027826:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763334.027839:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763334.027872:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496224768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763334.027875:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496225152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763334.027950:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763334.027960:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763334.027968:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496224768:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763334.028016:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496224896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763334.028058:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763334.028092:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763334/real 1789763334] req@ffff8dd205266680 x1876699496224896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763334.028117:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496224896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763334.028135:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763334.028142:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763334.028150:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496224896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763334.028163:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496225024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763334.028304:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763334.028325:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763334/real 1789763334] req@ffff8dd205267480 x1876699496225024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763334.028342:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496225024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763334.028353:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763334.028358:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763334.028362:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496225024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763334.028370:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496225152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763334.028395:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763334.028407:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763334/real 1789763334] req@ffff8dd205264a80 x1876699496225152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763334.028421:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496225152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763334.028432:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763334.028435:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763334.028439:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496225152:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763334.603406:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763335.627421:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763336.651417:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763337.675298:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763338.699391:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763339.147393:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763339.147409:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763339.147416:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763339.147431:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3318 00000100:00080000:1.0:1789763339.147463:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763339.147474:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763339.147534:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496225280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763339.147576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763339.147582:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763339.147585:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763339.147590:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3318 00000100:00080000:1.0:1789763339.147595:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763339.147600:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763339.147614:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763339.147637:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496225408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763339.147647:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496225280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763339.147658:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763339.147664:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763339.147666:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763339.147673:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3318 00000100:00080000:1.0:1789763339.147680:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763339.147686:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763339.147695:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763339.147710:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496225536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763339.147724:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789763339.147735:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763339.147741:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763339.147744:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763339.147751:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3318 00000100:00080000:1.0:1789763339.147759:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763339.147764:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763339.147773:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763339.147795:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496225664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763339.147796:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763339/real 1789763339] req@ffff8dd20504e680 x1876699496225280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763339.147884:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496225280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763339.147897:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763339.147902:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763339.147908:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496225280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763339.147931:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496225408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763339.147949:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763339.147958:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763339/real 1789763339] req@ffff8dd20504df80 x1876699496225408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763339.147966:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496225408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763339.147972:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763339.147974:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763339.147976:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496225408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763339.147980:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496225536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763339.147991:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763339.147997:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763339/real 1789763339] req@ffff8dd20504d500 x1876699496225536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763339.148004:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496225536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763339.148014:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763339.148016:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763339.148018:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496225536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763339.148021:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496225664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763339.148032:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763339.148038:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763339/real 1789763339] req@ffff8dd20504e300 x1876699496225664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763339.148044:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496225664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763339.148050:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763339.148051:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763339.148053:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496225664:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763339.723444:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763340.747414:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763341.771435:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763342.795416:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763343.819449:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763344.267360:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763344.267421:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763344.267426:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763344.267437:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3323 00000100:00080000:2.0:1789763344.267450:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763344.267458:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763344.267508:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496225792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763344.267552:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763344.267559:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763344.267563:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763344.267570:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3323 00000100:00080000:2.0:1789763344.267577:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763344.267601:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763344.267616:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763344.267639:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496225920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763344.267648:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496225792:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763344.267669:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763344.267689:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763344.267693:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763344.267700:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3323 00000100:00080000:2.0:1789763344.267722:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763344.267729:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763344.267736:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763344.267752:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496226048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763344.267768:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763344.267772:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763344.267775:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763344.267781:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3323 00000100:00080000:2.0:1789763344.267787:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763344.267793:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763344.267803:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763344.267808:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496225920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763344.267838:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496226176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763344.267884:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496226048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763344.267925:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496226176:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763344.843465:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763345.867418:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763346.891340:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763347.339446:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789763347.339484:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763347.339500:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763347.339550:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763347.339560:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763347.339570:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789763347.339626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763344/real 1789763347] req@ffff8dd205267100 x1876699496226176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763347.339659:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496226176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763347.339671:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763347.339675:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763347.339679:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496226176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763347.339689:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763344/real 1789763347] req@ffff8dd205264380 x1876699496225920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763347.339706:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496225920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763347.339717:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763347.339722:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763347.339726:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496225920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763347.339734:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763344/real 1789763347] req@ffff8dd205264e00 x1876699496226048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763347.339764:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496226048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763347.339775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763347.339779:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763347.339784:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496226048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763347.339793:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763344/real 1789763347] req@ffff8dd205267800 x1876699496225792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763347.339810:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496225792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763347.339821:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763347.339825:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763347.339829:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496225792:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789763347.915335:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763348.939356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763349.387297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763349.387312:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763349.387317:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763349.387329:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3328 00000100:00080000:1.0:1789763349.387341:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763349.387350:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763349.387388:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496226304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763349.387418:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763349.387422:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763349.387424:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763349.387429:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3328 00000100:00080000:1.0:1789763349.387433:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763349.387437:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763349.387447:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763349.387461:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496226432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763349.387475:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763349.387481:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496226304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763349.387482:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763349.387487:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763349.387499:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3328 00000100:00080000:1.0:1789763349.387502:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763349.387505:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763349.387511:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763349.387529:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496226560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763349.387544:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763349.387549:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763349.387553:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763349.387559:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3328 00000100:00080000:1.0:1789763349.387565:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763349.387571:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763349.387578:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763349.387594:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763349.387600:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496226688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763349.387650:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763349/real 1789763349] req@ffff8dd20504d180 x1876699496226304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763349.387672:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496226304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763349.387684:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763349.387691:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763349.387697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496226304:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763349.387744:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496226432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763349.387782:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763349.387803:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763349/real 1789763349] req@ffff8dd20504ce00 x1876699496226432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763349.387824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496226432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763349.387837:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763349.387842:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763349.387848:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496226432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763349.387858:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496226560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763349.387890:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763349.387908:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763349/real 1789763349] req@ffff8dd20504fb80 x1876699496226560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763349.387935:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496226560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763349.387950:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763349.387955:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763349.387961:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496226560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763349.387969:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496226688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763349.388002:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763349.388021:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763349/real 1789763349] req@ffff8dd20504f800 x1876699496226688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763349.388038:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496226688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763349.388049:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763349.388054:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763349.388058:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496226688:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763349.963320:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763350.987315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763352.011292:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763353.035367:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763354.059390:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763354.507322:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763354.507334:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763354.507339:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763354.507350:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3334 00000100:00080000:0.0:1789763354.507361:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763354.507369:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763354.507411:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496226816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763354.507443:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763354.507447:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763354.507449:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763354.507454:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3334 00000100:00080000:0.0:1789763354.507457:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763354.507461:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763354.507471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763354.507485:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496226944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763354.507498:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763354.507506:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789763354.507506:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496226816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763354.507519:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763354.507525:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3334 00000100:00080000:0.0:1789763354.507531:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763354.507536:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763354.507542:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763354.507561:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496227072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763354.507580:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763354.507586:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763354.507589:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763354.507598:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763354.507604:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3334 00000100:00080000:0.0:1789763354.507614:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763354.507621:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763354.507631:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763354.507654:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496227200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763354.507667:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763354/real 1789763354] req@ffff8dd211837b80 x1876699496226816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763354.507693:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496226816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763354.507705:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763354.507709:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763354.507713:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496226816:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763354.507738:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496226944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763354.507781:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763354.507794:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763354/real 1789763354] req@ffff8dd211837800 x1876699496226944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763354.507806:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496226944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763354.507816:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763354.507819:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763354.507822:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496226944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763354.507827:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496227072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763354.507844:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763354.507853:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763354/real 1789763354] req@ffff8dd211836300 x1876699496227072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763354.507864:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496227072/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763354.507879:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763354.507882:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763354.507885:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496227072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763354.507890:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496227200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763354.507905:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763354.507913:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763354/real 1789763354] req@ffff8dd211837100 x1876699496227200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763354.507923:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496227200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763354.507930:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763354.507933:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763354.507936:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496227200:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763355.083371:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763356.107344:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763357.131367:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763358.155367:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763359.179348:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763359.627303:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763359.627316:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763359.627323:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763359.627336:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3339 00000100:00080000:1.0:1789763359.627352:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763359.627362:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763359.627412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496227328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763359.627446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763359.627451:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763359.627453:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763359.627459:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3339 00000100:00080000:1.0:1789763359.627462:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763359.627466:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763359.627478:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763359.627495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496227456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763359.627509:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763359.627514:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763359.627517:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763359.627522:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3339 00000100:00080000:1.0:1789763359.627527:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763359.627536:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496227328:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763359.627538:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763359.627554:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763359.627573:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496227584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763359.627588:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763359.627594:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763359.627596:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763359.627602:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3339 00000100:00080000:1.0:1789763359.627607:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763359.627611:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763359.627617:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763359.627634:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496227712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763359.627641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763359.627708:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763359/real 1789763359] req@ffff8dd20504c700 x1876699496227328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763359.627733:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496227328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763359.627746:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763359.627752:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763359.627757:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496227328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763359.627782:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496227456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763359.627815:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763359.627840:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763359/real 1789763359] req@ffff8dd20504f100 x1876699496227456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763359.627867:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496227456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763359.627885:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763359.627893:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763359.627900:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496227456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763359.627954:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496227584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763359.627987:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763359.628002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763359/real 1789763359] req@ffff8dd20504ed80 x1876699496227584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763359.628018:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496227584/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763359.628029:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763359.628033:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763359.628038:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496227584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763359.628044:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496227712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763359.628064:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763359.628119:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763359/real 1789763359] req@ffff8dd20504d880 x1876699496227712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763359.628146:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496227712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763359.628164:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763359.628214:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763359.628220:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496227712:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763360.203415:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763361.227398:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763362.251364:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763363.275400:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763364.299389:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763364.747296:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763364.747309:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763364.747315:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763364.747328:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3344 00000100:00080000:1.0:1789763364.747341:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763364.747351:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763364.747417:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496227840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763364.747467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763364.747475:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763364.747479:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763364.747487:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3344 00000100:00080000:1.0:1789763364.747495:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763364.747503:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763364.747518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763364.747551:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496227968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763364.747565:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496227840:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763364.747591:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763364.747599:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763364.747604:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763364.747614:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3344 00000100:00080000:1.0:1789763364.747623:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763364.747631:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763364.747640:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763364.747665:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496228096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763364.747667:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763364.747688:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763364.747693:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763364.747697:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763364.747702:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3344 00000100:00080000:1.0:1789763364.747708:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763364.747713:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763364.747719:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763364.747733:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496228224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763364.747751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763364/real 1789763364] req@ffff8dd20504c000 x1876699496227840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763364.747785:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496227840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763364.747801:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763364.747807:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763364.747812:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496227840:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763364.747843:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496227968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763364.747883:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763364.747899:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763364/real 1789763364] req@ffff8dd20504ca80 x1876699496227968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763364.747927:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496227968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763364.747946:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763364.747950:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763364.747954:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496227968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763364.747961:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496228096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763364.747982:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763364.747993:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763364/real 1789763364] req@ffff8dd20504ea00 x1876699496228096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763364.748007:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496228096/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763364.748017:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763364.748020:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763364.748024:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496228096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763364.748032:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496228224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763364.748105:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763364.748129:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763364/real 1789763364] req@ffff8dd20504c380 x1876699496228224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763364.748154:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496228224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763364.748210:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763364.748215:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763364.748221:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496228224:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763365.323295:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763366.347304:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763367.371406:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763368.395414:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763369.419290:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763369.867242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763369.867251:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763369.867255:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763369.867263:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3349 00000100:00080000:3.0:1789763369.867273:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763369.867279:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763369.867311:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496228352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763369.867333:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763369.867336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763369.867337:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763369.867341:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3349 00000100:00080000:3.0:1789763369.867343:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763369.867345:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763369.867353:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763369.867363:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496228480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763369.867371:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763369.867373:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763369.867374:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763369.867379:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3349 00000100:00080000:3.0:1789763369.867381:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763369.867382:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496228352:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763369.867387:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763369.867394:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763369.867403:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496228608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763369.867411:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763369.867413:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763369.867414:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763369.867417:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3349 00000100:00080000:3.0:1789763369.867419:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763369.867421:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763369.867424:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763369.867432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496228736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763369.867457:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763369.867496:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763369/real 1789763369] req@ffff8dd203144380 x1876699496228352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763369.867511:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496228352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763369.867518:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763369.867521:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763369.867524:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496228352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763369.867543:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496228480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763369.867560:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763369.867568:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763369/real 1789763369] req@ffff8dd203146d80 x1876699496228480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763369.867577:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496228480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763369.867583:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763369.867585:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763369.867587:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496228480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763369.867591:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496228608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763369.867602:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763369.867607:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763369/real 1789763369] req@ffff8dd203147b80 x1876699496228608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763369.867615:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496228608/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763369.867620:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763369.867622:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763369.867624:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496228608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763369.867627:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496228736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763369.867637:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763369.867643:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763369/real 1789763369] req@ffff8dd203146a00 x1876699496228736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763369.867650:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496228736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763369.867660:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763369.867662:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763369.867663:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496228736:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763370.443281:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763371.467427:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763372.491288:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763373.515363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763374.539423:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763374.987323:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763374.987341:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763374.987349:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763374.987366:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3354 00000100:00080000:1.0:1789763374.987384:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763374.987397:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763374.987455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496228864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763374.987497:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763374.987502:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763374.987505:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763374.987510:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3354 00000100:00080000:1.0:1789763374.987515:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763374.987520:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763374.987532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763374.987553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496228992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763374.987568:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763374.987572:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763374.987575:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763374.987580:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3354 00000100:00080000:1.0:1789763374.987584:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763374.987593:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496228864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763374.987594:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763374.987607:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763374.987620:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496229120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763374.987636:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763374.987642:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763374.987644:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763374.987650:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3354 00000100:00080000:1.0:1789763374.987654:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763374.987659:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763374.987665:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763374.987678:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496229248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763374.987711:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763374.987781:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763374/real 1789763374] req@ffff8dd20504dc00 x1876699496228864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763374.987817:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496228864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763374.987839:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763374.987847:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763374.987856:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496228864:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763374.987867:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496228992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763374.987904:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763374.987922:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763374/real 1789763374] req@ffff8dd20504c380 x1876699496228992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763374.987938:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496228992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763374.987957:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763374.987962:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763374.987992:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496228992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763374.988029:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496229120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763374.988068:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763374.988084:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763374/real 1789763374] req@ffff8dd20504ea00 x1876699496229120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763374.988099:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496229120/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763374.988110:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763374.988114:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763374.988118:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496229120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763374.988124:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496229248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763374.988144:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763374.988155:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763374/real 1789763374] req@ffff8dd20504ca80 x1876699496229248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763374.988206:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496229248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763374.988216:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763374.988220:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763374.988226:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496229248:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763375.563385:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763376.587301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763377.611398:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763378.635379:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763379.659464:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763380.107342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763380.107355:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763380.107361:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763380.107374:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3359 00000100:00080000:3.0:1789763380.107388:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763380.107397:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763380.107452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496229376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763380.107502:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763380.107509:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763380.107514:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763380.107523:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3359 00000100:00080000:3.0:1789763380.107532:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763380.107541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763380.107559:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763380.107580:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496229376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763380.107590:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496229504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763380.107609:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763380.107613:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763380.107616:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763380.107621:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3359 00000100:00080000:3.0:1789763380.107627:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763380.107633:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763380.107641:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763380.107656:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496229632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763380.107671:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789763380.107677:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763380.107680:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763380.107686:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763380.107693:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3359 00000100:00080000:3.0:1789763380.107699:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763380.107704:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763380.107711:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763380.107728:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496229760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763380.107746:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763380/real 1789763380] req@ffff8dd203144e00 x1876699496229376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763380.107776:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496229376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763380.107792:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763380.107797:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763380.107802:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496229376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763380.107832:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496229504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763380.107861:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763380.107876:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763380/real 1789763380] req@ffff8dd203146a00 x1876699496229504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763380.107909:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496229504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763380.107918:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763380.107922:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763380.107925:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496229504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763380.107961:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496229632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763380.107986:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763380.107999:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763380/real 1789763380] req@ffff8dd203147b80 x1876699496229632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763380.108012:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496229632/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763380.108021:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763380.108024:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763380.108028:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496229632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763380.108033:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496229760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763380.108050:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763380.108060:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763380/real 1789763380] req@ffff8dd203146d80 x1876699496229760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763380.108071:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496229760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763380.108080:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763380.108083:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763380.108086:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496229760:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763380.683378:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763381.707416:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763382.731438:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763383.755361:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763384.779376:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763385.227342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763385.227355:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763385.227361:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763385.227373:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3364 00000100:00080000:0.0:1789763385.227393:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763385.227403:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763385.227452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496229888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763385.227487:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763385.227492:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763385.227495:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763385.227500:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3364 00000100:00080000:0.0:1789763385.227504:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763385.227508:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763385.227520:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763385.227535:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496230016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763385.227548:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763385.227553:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763385.227556:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763385.227561:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3364 00000100:00080000:0.0:1789763385.227565:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763385.227570:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763385.227577:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763385.227586:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496229888:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763385.227596:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496230144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763385.227617:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763385.227621:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763385.227624:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763385.227629:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3364 00000100:00080000:0.0:1789763385.227635:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763385.227640:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763385.227647:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763385.227663:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496230272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763385.227690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763385.227745:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763385/real 1789763385] req@ffff8dd211836a00 x1876699496229888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763385.227770:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496229888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763385.227785:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763385.227790:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763385.227796:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496229888:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763385.227806:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496230016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763385.227831:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763385.227846:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763385/real 1789763385] req@ffff8dd211834000 x1876699496230016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763385.227861:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496230016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763385.227872:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763385.227875:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763385.227879:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496230016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763385.227934:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496230144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763385.227960:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763385.227974:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763385/real 1789763385] req@ffff8dd211834700 x1876699496230144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763385.227998:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496230144/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763385.228008:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763385.228012:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763385.228016:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496230144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763385.228022:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496230272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763385.228040:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763385.228051:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763385/real 1789763385] req@ffff8dd211835c00 x1876699496230272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763385.228065:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496230272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763385.228075:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763385.228078:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763385.228082:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496230272:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763385.803391:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763386.827336:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763387.851395:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763388.875301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763389.899518:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763390.347367:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763390.347384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763390.347392:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763390.347410:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3369 00000100:00080000:0.0:1789763390.347430:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763390.347445:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763390.347510:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496230400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763390.347548:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763390.347554:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763390.347557:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763390.347562:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3369 00000100:00080000:0.0:1789763390.347568:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763390.347573:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763390.347588:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763390.347606:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496230528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763390.347623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496230400:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763390.347629:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763390.347649:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763390.347652:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763390.347659:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3369 00000100:00080000:0.0:1789763390.347663:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763390.347670:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763390.347678:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763390.347695:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496230656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763390.347712:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763390.347716:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763390.347719:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763390.347725:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3369 00000100:00080000:0.0:1789763390.347730:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763390.347735:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763390.347741:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763390.347743:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763390.347773:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496230784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763390.347837:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763390/real 1789763390] req@ffff8dd211835f80 x1876699496230400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763390.347890:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496230400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763390.347906:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763390.347913:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763390.347918:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496230400:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763390.347964:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496230528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763390.348007:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763390.348025:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763390/real 1789763390] req@ffff8dd211834380 x1876699496230528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763390.348041:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496230528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763390.348053:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763390.348059:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763390.348065:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496230528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763390.348078:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496230656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763390.348117:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763390.348142:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763390/real 1789763390] req@ffff8dd211836680 x1876699496230656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763390.348211:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496230656/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763390.348224:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763390.348229:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763390.348234:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496230656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763390.348243:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496230784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763390.348276:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763390.348292:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763390/real 1789763390] req@ffff8dd211834a80 x1876699496230784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763390.348308:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496230784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763390.348318:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763390.348322:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763390.348326:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496230784:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763390.923353:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763391.947399:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763392.971381:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763393.995448:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763395.019395:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763395.467325:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763395.467338:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763395.467344:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763395.467368:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3375 00000100:00080000:0.0:1789763395.467381:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763395.467390:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763395.467447:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496230912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763395.467496:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763395.467504:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763395.467508:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763395.467516:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3375 00000100:00080000:0.0:1789763395.467524:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763395.467533:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763395.467549:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763395.467576:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496231040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763395.467588:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496230912:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763395.467600:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763395.467610:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763395.467614:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763395.467624:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3375 00000100:00080000:0.0:1789763395.467632:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763395.467648:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763395.467661:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763395.467696:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496231168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763395.467696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763395.467729:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763395.467734:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763395.467737:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763395.467747:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3375 00000100:00100000:2.0:1789763395.467756:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763395/real 1789763395] req@ffff8dd211836d80 x1876699496230912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763395.467759:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763395.467795:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763395.467798:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496230912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763395.467843:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789763395.467850:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763395.467874:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763395.467893:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496230912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763395.467895:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496231296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763395.467943:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496231040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763395.467984:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763395.468003:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763395/real 1789763395] req@ffff8dd211835180 x1876699496231040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763395.468020:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496231040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763395.468031:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763395.468036:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763395.468040:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496231040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763395.468047:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496231168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763395.468078:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763395.468090:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763395/real 1789763395] req@ffff8dd211837480 x1876699496231168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763395.468104:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496231168/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763395.468115:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763395.468119:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763395.468122:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496231168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763395.468129:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496231296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763395.468150:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763395.468161:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763395/real 1789763395] req@ffff8dd211834e00 x1876699496231296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763395.468230:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496231296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763395.468240:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763395.468244:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763395.468249:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496231296:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763396.043405:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763397.067376:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763398.091419:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763399.115364:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763400.139395:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763400.587388:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763400.587402:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763400.587408:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763400.587422:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3380 00000100:00080000:3.0:1789763400.587439:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763400.587451:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763400.587530:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496231424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763400.587568:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763400.587574:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763400.587577:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763400.587583:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3380 00000100:00080000:3.0:1789763400.587589:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763400.587594:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763400.587606:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763400.587631:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496231552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763400.587647:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763400.587652:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763400.587654:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763400.587662:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3380 00000100:00080000:3.0:1789763400.587673:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763400.587675:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496231424:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763400.587683:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763400.587701:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763400.587715:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496231680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763400.587731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763400.587737:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763400.587740:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763400.587745:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3380 00000100:00080000:3.0:1789763400.587749:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763400.587756:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763400.587761:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763400.587776:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496231808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763400.587856:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763400.587934:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763400/real 1789763400] req@ffff8dd203144380 x1876699496231424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763400.587959:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496231424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763400.587973:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763400.587978:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763400.587982:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496231424:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763400.588016:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496231552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763400.588044:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763400.588058:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763400/real 1789763400] req@ffff8dd203145500 x1876699496231552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763400.588078:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496231552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763400.588086:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763400.588089:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763400.588092:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496231552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763400.588098:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496231680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763400.588117:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763400.588126:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763400/real 1789763400] req@ffff8dd203144a80 x1876699496231680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763400.588138:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496231680/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763400.588145:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763400.588148:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763400.588150:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496231680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763400.588156:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496231808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763400.588171:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763400.588180:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763400/real 1789763400] req@ffff8dd203145f80 x1876699496231808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763400.588245:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496231808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763400.588260:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763400.588264:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763400.588267:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496231808:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763401.163407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763402.187363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763403.211387:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763404.235362:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763405.259364:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763405.707406:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763405.707419:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763405.707424:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763405.707436:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3385 00000100:00080000:2.0:1789763405.707449:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763405.707457:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763405.707514:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496231936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763405.707551:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763405.707555:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763405.707558:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763405.707562:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3385 00000100:00080000:2.0:1789763405.707584:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763405.707590:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763405.707603:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763405.707620:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496232064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763405.707631:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496231936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763405.707643:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763405.707650:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763405.707653:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763405.707659:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3385 00000100:00080000:2.0:1789763405.707664:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763405.707669:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763405.707675:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763405.707691:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496232192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763405.707705:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763405.707709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763405.707711:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763405.707717:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3385 00000100:00080000:2.0:1789763405.707723:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763405.707727:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763405.707734:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763405.707767:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763405.707782:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496232320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763405.707889:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763405/real 1789763405] req@ffff8dd205264000 x1876699496231936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763405.707917:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496231936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763405.707932:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763405.707937:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763405.707942:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496231936:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763405.707976:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496232064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763405.708019:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763405.708036:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763405/real 1789763405] req@ffff8dd205265880 x1876699496232064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763405.708052:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496232064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763405.708063:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763405.708067:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763405.708072:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496232064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763405.708078:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496232192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763405.708098:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763405.708110:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763405/real 1789763405] req@ffff8dd205267b80 x1876699496232192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763405.708124:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496232192/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763405.708134:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763405.708137:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763405.708141:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496232192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763405.708147:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496232320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763405.708165:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763405.708194:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763405/real 1789763405] req@ffff8dd205266a00 x1876699496232320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763405.708208:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496232320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763405.708247:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763405.708250:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763405.708254:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496232320:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763406.283331:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763407.307356:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763408.331333:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763409.355391:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763410.379421:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763410.827348:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763410.827362:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763410.827367:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763410.827380:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3390 00000100:00080000:3.0:1789763410.827396:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763410.827410:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763410.827470:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496232448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763410.827524:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763410.827530:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763410.827532:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763410.827539:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3390 00000100:00080000:3.0:1789763410.827545:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763410.827551:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763410.827564:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763410.827591:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496232448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763410.827592:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496232576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763410.827632:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763410.827638:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763410.827642:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763410.827652:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3390 00000100:00080000:3.0:1789763410.827674:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763410.827683:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763410.827694:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763410.827748:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496232704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763410.827769:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763410.827775:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763410.827778:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763410.827786:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496232576:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1789763410.827790:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3390 00000100:00080000:3.0:1789763410.827802:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763410.827811:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763410.827820:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763410.827843:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496232832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763410.827875:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496232704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763410.827928:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496232832:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763411.403442:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763412.427355:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763413.451429:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763413.899522:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789763413.899574:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763413.899594:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763413.899641:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763413.899652:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763413.899662:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1789763413.899822:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763410/real 1789763413] req@ffff8dd203147800 x1876699496232832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763413.899896:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496232832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763413.899923:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763413.899933:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763413.899942:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496232832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763413.899965:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763410/real 1789763413] req@ffff8dd203145c00 x1876699496232576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763413.899992:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496232576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763413.900011:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763413.900019:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763413.900027:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496232576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763413.900041:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763410/real 1789763413] req@ffff8dd203146300 x1876699496232704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763413.900065:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496232704/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763413.900081:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763413.900089:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763413.900095:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496232704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763413.900109:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763410/real 1789763413] req@ffff8dd203144700 x1876699496232448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763413.900135:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496232448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763413.900152:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763413.900159:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763413.900235:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496232448:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789763414.475308:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763415.499396:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763415.947252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763415.947265:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763415.947271:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763415.947289:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3395 00000100:00080000:3.0:1789763415.947309:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763415.947322:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763415.947400:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496232960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763415.947446:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763415.947453:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763415.947457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763415.947465:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3395 00000100:00080000:3.0:1789763415.947473:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763415.947483:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763415.947499:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763415.947514:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496232960:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763415.947531:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496233088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763415.947549:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763415.947554:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763415.947571:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763415.947585:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3395 00000100:00080000:3.0:1789763415.947594:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763415.947602:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763415.947613:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763415.947638:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496233216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763415.947644:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763415.947693:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763415.947709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763415.947713:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763415.947723:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3395 00000100:00080000:3.0:1789763415.947732:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763415.947739:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763415/real 1789763415] req@ffff8dd203145180 x1876699496232960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763415.947748:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763415.947767:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763415.947789:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496232960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763415.947803:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496233344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763415.947821:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763415.947835:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763415.947846:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496232960:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763415.947898:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496233088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763415.947948:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763415.947975:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763415/real 1789763415] req@ffff8dd203144000 x1876699496233088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763415.948013:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496233088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763415.948029:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763415.948035:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763415.948042:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496233088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763415.948053:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496233216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763415.948091:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763415.948113:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763415/real 1789763415] req@ffff8dd203147100 x1876699496233216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763415.948136:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496233216/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763415.948150:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763415.948156:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763415.948163:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496233216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763415.948210:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496233344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763415.948256:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763415.948280:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763415/real 1789763415] req@ffff8dd203145880 x1876699496233344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763415.948304:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496233344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763415.948332:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763415.948339:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763415.948347:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496233344:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763416.523446:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763417.547475:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763418.571394:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763419.595348:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763420.619411:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763421.067229:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763421.067239:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763421.067244:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763421.067254:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3400 00000100:00080000:3.0:1789763421.067269:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763421.067279:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763421.067325:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496233472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763421.067354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763421.067358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763421.067360:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763421.067364:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3400 00000100:00080000:3.0:1789763421.067368:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763421.067372:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763421.067382:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763421.067401:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496233472:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763421.067402:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496233600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763421.067414:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763421.067420:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763421.067422:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763421.067425:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3400 00000100:00080000:3.0:1789763421.067428:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763421.067431:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763421.067436:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763421.067452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496233728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763421.067461:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763421.067465:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763421.067467:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763421.067470:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3400 00000100:00080000:3.0:1789763421.067474:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763421.067478:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763421.067482:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763421.067489:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763421.067495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496233856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763421.067539:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763421/real 1789763421] req@ffff8dd203147480 x1876699496233472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763421.067559:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496233472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763421.067575:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763421.067581:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763421.067587:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496233472:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763421.067618:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496233600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763421.067671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763421.067701:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763421/real 1789763421] req@ffff8dd203145880 x1876699496233600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763421.067716:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496233600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763421.067725:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763421.067728:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763421.067731:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496233600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763421.067738:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496233728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763421.067756:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763421.067765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763421/real 1789763421] req@ffff8dd203147100 x1876699496233728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763421.067776:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496233728/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763421.067783:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763421.067785:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763421.067788:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496233728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763421.067792:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496233856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763421.067806:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763421.067814:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763421/real 1789763421] req@ffff8dd203144000 x1876699496233856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763421.067824:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496233856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763421.067831:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763421.067833:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763421.067836:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496233856:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763421.643412:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763422.667461:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763423.691489:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763424.715462:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763425.739450:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763426.187351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763426.187368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763426.187376:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763426.187393:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3405 00000100:00080000:0.0:1789763426.187411:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763426.187423:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763426.187493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496233984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763426.187550:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763426.187556:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763426.187559:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763426.187564:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3405 00000100:00080000:0.0:1789763426.187569:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763426.187574:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763426.187607:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763426.187630:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496234112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763426.187655:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763426.187660:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763426.187667:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763426.187670:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496233984:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763426.187676:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3405 00000100:00080000:0.0:1789763426.187689:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763426.187694:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763426.187702:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763426.187719:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496234240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763426.187732:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763426.187737:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763426.187739:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763426.187743:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3405 00000100:00080000:0.0:1789763426.187748:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763426.187752:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763426.187758:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763426.187771:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496234368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763426.187808:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763426.187878:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763426/real 1789763426] req@ffff8dd211834e00 x1876699496233984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763426.187905:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496233984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763426.187925:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763426.187933:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763426.187942:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496233984:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763426.187990:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496234112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763426.188038:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763426.188058:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763426/real 1789763426] req@ffff8dd211837480 x1876699496234112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763426.188076:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496234112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763426.188087:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763426.188092:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763426.188097:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496234112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763426.188105:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496234240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763426.188129:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763426.188141:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763426/real 1789763426] req@ffff8dd211835180 x1876699496234240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763426.188206:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496234240/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763426.188219:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763426.188223:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763426.188228:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496234240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763426.188235:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496234368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763426.188262:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763426.188276:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763426/real 1789763426] req@ffff8dd211836d80 x1876699496234368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763426.188292:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496234368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763426.188302:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763426.188306:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763426.188311:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496234368:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763426.763404:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763427.787451:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763428.811440:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763429.835485:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763430.859409:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763431.307359:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763431.307372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763431.307379:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763431.307395:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3410 00000100:00080000:3.0:1789763431.307415:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763431.307426:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763431.307487:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496234496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763431.307530:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763431.307536:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763431.307538:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763431.307544:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3410 00000100:00080000:3.0:1789763431.307549:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763431.307584:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763431.307607:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763431.307632:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496234624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763431.307654:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763431.307661:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763431.307665:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763431.307674:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3410 00000100:00080000:3.0:1789763431.307683:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763431.307697:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763431.307701:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496234496:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1789763431.307711:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763431.307743:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496234752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763431.307767:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763431.307775:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763431.307779:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763431.307787:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3410 00000100:00080000:3.0:1789763431.307800:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763431.307810:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763431.307821:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763431.307828:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763431.307857:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496234880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763431.307909:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763431/real 1789763431] req@ffff8dd203145180 x1876699496234496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763431.307956:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496234496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763431.307974:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763431.307980:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763431.307986:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496234496:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763431.307997:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496234624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763431.308028:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763431.308045:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763431/real 1789763431] req@ffff8dd203144700 x1876699496234624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763431.308061:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496234624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763431.308072:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763431.308076:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763431.308081:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496234624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763431.308115:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496234752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763431.308157:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763431.308208:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763431/real 1789763431] req@ffff8dd203146300 x1876699496234752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763431.308225:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496234752/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763431.308237:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763431.308241:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763431.308245:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496234752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763431.308253:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496234880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763431.308279:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763431.308292:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763431/real 1789763431] req@ffff8dd203145c00 x1876699496234880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763431.308307:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496234880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763431.308317:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763431.308321:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763431.308325:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496234880:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763431.883413:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763432.907391:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763433.931363:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763434.955400:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763435.979455:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763436.427355:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763436.427368:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763436.427373:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763436.427385:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3416 00000100:00080000:2.0:1789763436.427398:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763436.427406:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763436.427457:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496235008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763436.427502:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763436.427510:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763436.427533:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763436.427543:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3416 00000100:00080000:2.0:1789763436.427549:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763436.427556:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763436.427568:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496235008:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789763436.427569:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763436.427596:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496235136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763436.427611:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763436.427615:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763436.427618:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763436.427622:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3416 00000100:00080000:2.0:1789763436.427626:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763436.427630:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763436.427637:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763436.427650:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496235264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763436.427663:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789763436.427664:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763436.427673:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763436.427677:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763436.427683:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3416 00000100:00080000:2.0:1789763436.427688:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763436.427693:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763436.427699:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763436.427715:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496235392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763436.427727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763436/real 1789763436] req@ffff8dd205264700 x1876699496235008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763436.427752:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496235008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763436.427766:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763436.427771:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763436.427775:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496235008:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763436.427800:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496235136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763436.427824:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763436.427837:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763436/real 1789763436] req@ffff8dd205265f80 x1876699496235136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763436.427850:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496235136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763436.427860:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763436.427864:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763436.427868:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496235136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763436.427873:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496235264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763436.427890:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763436.427900:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763436/real 1789763436] req@ffff8dd205265180 x1876699496235264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763436.427912:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496235264/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763436.427921:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763436.427924:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763436.427927:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496235264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763436.427933:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496235392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763436.427949:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763436.427958:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763436/real 1789763436] req@ffff8dd205266d80 x1876699496235392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763436.427970:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496235392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763436.427979:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763436.427982:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763436.427986:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496235392:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763437.003320:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763438.027433:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763439.051405:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763440.075471:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763441.099371:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763441.547312:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763441.547327:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763441.547334:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763441.547350:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3421 00000100:00080000:2.0:1789763441.547367:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763441.547380:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763441.547440:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496235520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763441.547505:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763441.547513:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763441.547518:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763441.547527:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3421 00000100:00080000:2.0:1789763441.547550:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763441.547559:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763441.547568:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496235520:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789763441.547582:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763441.547618:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496235648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763441.547640:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763441.547649:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763441.547653:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763441.547659:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763441.547669:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3421 00000100:00080000:2.0:1789763441.547681:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763441.547689:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763441.547702:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763441.547727:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496235776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763441.547736:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763441/real 1789763441] req@ffff8dd205266300 x1876699496235520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763441.547775:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763441.547784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1789763441.547787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496235520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763441.547790:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763441.547811:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3421 00000100:00080000:3.0:1789763441.547816:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763441.547821:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763441.547828:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1789763441.547840:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763441.547841:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496235520:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789763441.547860:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763441.547885:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496235648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763441.547892:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496235904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763441.547925:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763441.547947:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763441/real 1789763441] req@ffff8dd205266d80 x1876699496235648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763441.547964:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496235648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763441.547975:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763441.547979:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763441.547983:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496235648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763441.547991:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496235776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763441.548012:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763441.548025:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763441/real 1789763441] req@ffff8dd205265180 x1876699496235776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763441.548040:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496235776/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763441.548049:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763441.548053:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763441.548056:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496235776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763441.548072:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496235904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763441.548092:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763441.548104:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763441/real 1789763441] req@ffff8dd205265f80 x1876699496235904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763441.548118:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496235904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763441.548128:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763441.548131:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763441.548135:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496235904:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763442.123365:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763443.147422:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763444.171398:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763445.195402:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763446.219503:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763446.667397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763446.667410:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763446.667416:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763446.667429:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3426 00000100:00080000:0.0:1789763446.667476:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763446.667489:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763446.667546:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496236032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763446.667588:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763446.667593:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763446.667602:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763446.667608:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3426 00000100:00080000:0.0:1789763446.667613:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763446.667618:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763446.667631:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763446.667648:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496236160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763446.667672:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496236032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763446.667678:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763446.667685:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763446.667688:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763446.667695:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3426 00000100:00080000:0.0:1789763446.667699:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763446.667704:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763446.667711:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763446.667728:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496236288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763446.667740:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763446.667745:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763446.667748:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763446.667752:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3426 00000100:00080000:0.0:1789763446.667758:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789763446.667765:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763446.667768:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763446.667778:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763446.667792:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496236416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763446.667848:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763446/real 1789763446] req@ffff8dd211834a80 x1876699496236032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763446.667874:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496236032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763446.667889:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763446.667894:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763446.667900:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496236032:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763446.667930:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496236160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763446.667961:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763446.667977:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763446/real 1789763446] req@ffff8dd211836680 x1876699496236160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763446.667993:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496236160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763446.668004:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763446.668008:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763446.668012:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496236160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763446.668020:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496236288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763446.668041:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763446.668052:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763446/real 1789763446] req@ffff8dd211834380 x1876699496236288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763446.668067:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496236288/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763446.668077:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763446.668081:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763446.668085:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496236288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763446.668093:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496236416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763446.668113:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763446.668124:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763446/real 1789763446] req@ffff8dd211835f80 x1876699496236416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763446.668139:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496236416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763446.668148:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763446.668152:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763446.668155:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496236416:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763447.243412:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763448.267381:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763449.291471:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763450.315329:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763451.339480:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763451.787335:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763451.787349:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763451.787354:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763451.787369:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3431 00000100:00080000:3.0:1789763451.787382:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763451.787390:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763451.787474:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496236544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763451.787512:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763451.787518:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763451.787521:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763451.787526:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3431 00000100:00080000:3.0:1789763451.787531:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763451.787538:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763451.787551:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763451.787570:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496236672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763451.787588:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496236544:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763451.787590:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763451.787621:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763451.787624:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763451.787630:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3431 00000100:00080000:3.0:1789763451.787635:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763451.787640:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763451.787646:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763451.787664:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496236800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763451.787678:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763451.787684:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763451.787686:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763451.787690:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3431 00000800:00000100:2.0:1789763451.787699:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763451.787708:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763451.787735:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763451.787743:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763451.787762:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496236928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763451.787777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763451/real 1789763451] req@ffff8dd203147800 x1876699496236544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763451.787805:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496236544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763451.787821:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763451.787827:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763451.787832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496236544:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763451.787861:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496236672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763451.787891:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763451.787907:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763451/real 1789763451] req@ffff8dd203145f80 x1876699496236672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763451.787923:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496236672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763451.787934:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763451.787938:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763451.787943:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496236672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763451.787950:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496236800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763451.787970:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763451.787981:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763451/real 1789763451] req@ffff8dd203144a80 x1876699496236800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763451.787996:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496236800/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763451.788006:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763451.788009:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763451.788013:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496236800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763451.788037:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496236928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763451.788057:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763451.788068:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763451/real 1789763451] req@ffff8dd203145500 x1876699496236928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763451.788082:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496236928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763451.788091:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763451.788111:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763451.788115:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496236928:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763452.363393:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763453.387430:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763454.411365:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763455.435350:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763456.459460:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763456.907308:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763456.907322:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763456.907327:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763456.907341:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3436 00000100:00080000:3.0:1789763456.907355:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763456.907412:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763456.907455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496237056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763456.907485:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763456.907490:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763456.907492:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763456.907496:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3436 00000100:00080000:3.0:1789763456.907500:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763456.907504:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763456.907514:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763456.907527:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496237184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763456.907539:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763456.907543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763456.907545:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763456.907549:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3436 00000100:00080000:3.0:1789763456.907552:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763456.907558:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496237056:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763456.907561:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763456.907574:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763456.907587:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496237312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763456.907600:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763456.907604:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763456.907607:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763456.907611:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3436 00000100:00080000:3.0:1789763456.907624:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763456.907629:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763456.907634:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763456.907641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763456.907651:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496237440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763456.907691:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763456/real 1789763456] req@ffff8dd203144380 x1876699496237056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763456.907713:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496237056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763456.907725:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763456.907729:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763456.907734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496237056:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763456.907741:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496237184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763456.907785:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763456.907817:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763456/real 1789763456] req@ffff8dd203146d80 x1876699496237184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763456.907834:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496237184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763456.907854:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763456.907859:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763456.907863:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496237184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763456.907902:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496237312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763456.907943:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763456.907969:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763456/real 1789763456] req@ffff8dd203147b80 x1876699496237312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763456.907993:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496237312/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763456.908005:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763456.908011:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763456.908018:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496237312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763456.908031:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496237440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763456.908071:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763456.908095:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763456/real 1789763456] req@ffff8dd203146a00 x1876699496237440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763456.908121:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496237440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763456.908131:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763456.908135:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763456.908141:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496237440:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763457.483456:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763458.507373:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763459.531322:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763460.555456:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763461.579507:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763462.027399:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763462.027411:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763462.027417:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763462.027431:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3441 00000100:00080000:2.0:1789763462.027445:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763462.027455:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763462.027500:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496237568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763462.027529:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763462.027534:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763462.027536:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763462.027541:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3441 00000100:00080000:2.0:1789763462.027544:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763462.027549:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763462.027560:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763462.027574:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496237696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763462.027586:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496237568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763462.027592:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763462.027599:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763462.027601:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763462.027606:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3441 00000100:00080000:2.0:1789763462.027613:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763462.027617:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763462.027623:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763462.027638:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496237824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763462.027660:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763462.027665:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763462.027667:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763462.027671:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3441 00000100:00080000:2.0:1789763462.027675:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789763462.027683:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763462.027685:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763462.027694:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763462.027707:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496237952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763462.027759:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763462/real 1789763462] req@ffff8dd205264700 x1876699496237568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763462.027794:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496237568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763462.027810:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763462.027815:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763462.027820:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496237568:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763462.027848:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496237696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763462.027880:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763462.027894:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763462/real 1789763462] req@ffff8dd205266a00 x1876699496237696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763462.027910:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496237696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763462.027919:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763462.027923:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763462.027926:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496237696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763462.027932:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496237824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763462.027950:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763462.027960:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763462/real 1789763462] req@ffff8dd205267b80 x1876699496237824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763462.027992:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496237824/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763462.028006:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763462.028012:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763462.028018:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496237824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763462.028028:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496237952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763462.028063:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763462.028081:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763462/real 1789763462] req@ffff8dd205265880 x1876699496237952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763462.028095:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496237952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763462.028111:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763462.028115:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763462.028119:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496237952:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763462.603496:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763463.627453:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763464.651436:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763465.675485:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763466.699424:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763467.147387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763467.147401:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763467.147407:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763467.147426:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3446 00000100:00080000:1.0:1789763467.147441:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763467.147450:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763467.147497:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496238080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763467.147534:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763467.147539:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763467.147541:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763467.147547:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3446 00000100:00080000:1.0:1789763467.147551:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763467.147555:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763467.147567:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763467.147586:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496238208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763467.147605:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763467.147610:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763467.147613:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763467.147623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496238080:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763467.147625:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3446 00000100:00080000:1.0:1789763467.147637:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763467.147646:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763467.147652:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763467.147666:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496238336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763467.147680:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763467.147685:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763467.147688:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763467.147692:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3446 00000100:00080000:1.0:1789763467.147696:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763467.147701:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763467.147706:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763467.147718:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496238464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763467.147745:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763467.147813:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763467/real 1789763467] req@ffff8dd20504c000 x1876699496238080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763467.147851:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496238080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763467.147870:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763467.147875:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763467.147881:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496238080:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763467.147913:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496238208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763467.147953:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763467.147977:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763467/real 1789763467] req@ffff8dd20504d880 x1876699496238208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763467.148004:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496238208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763467.148023:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763467.148030:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763467.148037:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496238208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763467.148046:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496238336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763467.148077:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763467.148092:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763467/real 1789763467] req@ffff8dd20504ed80 x1876699496238336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763467.148107:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496238336/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763467.148117:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763467.148121:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763467.148126:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496238336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763467.148132:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496238464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763467.148151:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763467.148162:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763467/real 1789763467] req@ffff8dd20504f100 x1876699496238464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763467.148233:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496238464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763467.148244:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763467.148248:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763467.148252:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496238464:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763467.723450:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763468.747308:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763469.771431:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763470.795308:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763471.819428:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763472.267300:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763472.267308:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763472.267312:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763472.267321:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3451 00000100:00080000:1.0:1789763472.267330:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763472.267336:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763472.267369:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496238592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763472.267388:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763472.267391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763472.267393:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763472.267396:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3451 00000100:00080000:1.0:1789763472.267398:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763472.267400:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763472.267407:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763472.267415:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496238720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763472.267421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763472.267423:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763472.267425:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763472.267427:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3451 00000100:00080000:1.0:1789763472.267428:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763472.267431:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763472.267433:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763472.267439:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496238848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763472.267445:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763472.267448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763472.267450:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763472.267452:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3451 00000100:00080000:1.0:1789763472.267455:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763472.267457:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763472.267461:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763472.267466:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496238592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763472.267471:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496238976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763472.267525:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763472.267569:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763472/real 1789763472] req@ffff8dd20504c700 x1876699496238592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763472.267583:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496238592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763472.267591:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763472.267594:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763472.267597:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496238592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763472.267602:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496238720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763472.267615:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763472.267622:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763472/real 1789763472] req@ffff8dd20504f800 x1876699496238720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763472.267630:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496238720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763472.267635:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763472.267637:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763472.267639:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496238720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763472.267643:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496238848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763472.267653:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763472.267658:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763472/real 1789763472] req@ffff8dd20504fb80 x1876699496238848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763472.267666:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496238848/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763472.267675:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763472.267677:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763472.267679:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496238848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763472.267699:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496238976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763472.267709:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763472.267715:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763472/real 1789763472] req@ffff8dd20504ce00 x1876699496238976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763472.267722:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496238976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763472.267727:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763472.267728:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763472.267730:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496238976:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763472.843418:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763473.867429:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763474.891490:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763475.915487:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763476.939421:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763477.387373:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763477.387385:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763477.387390:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763477.387402:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3456 00000100:00080000:0.0:1789763477.387414:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763477.387422:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763477.387487:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496239104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763477.387547:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763477.387552:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763477.387555:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763477.387561:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3456 00000100:00080000:0.0:1789763477.387567:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763477.387572:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763477.387596:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763477.387633:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496239232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763477.387646:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763477.387651:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763477.387653:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763477.387659:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3456 00000100:00080000:0.0:1789763477.387663:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763477.387669:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763477.387677:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763477.387698:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496239360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763477.387716:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496239104:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763477.387725:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763477.387737:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763477.387741:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763477.387750:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3456 00000100:00080000:0.0:1789763477.387759:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763477.387768:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763477.387781:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763477.387801:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496239488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763477.387879:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496239232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763477.387959:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496239360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763477.387995:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496239488:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763477.963402:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763478.987395:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763480.011405:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763480.459631:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789763480.459698:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763480.459745:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763480.459830:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763480.459846:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763480.459868:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789763480.459939:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763477/real 1789763480] req@ffff8dd211834000 x1876699496239360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763480.460003:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496239360/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763480.460028:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763480.460055:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763480.460063:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496239360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763480.460077:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763477/real 1789763480] req@ffff8dd211836a00 x1876699496239488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763480.460099:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496239488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763480.460113:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763480.460120:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763480.460126:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496239488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763480.460138:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763477/real 1789763480] req@ffff8dd211835c00 x1876699496239104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763480.460159:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496239104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763480.460173:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763480.460179:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763480.460184:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496239104:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763480.460265:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763477/real 1789763480] req@ffff8dd211834700 x1876699496239232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763480.460289:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496239232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763480.460306:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763480.460314:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763480.460322:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496239232:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763481.035429:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763482.059343:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763482.507412:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763482.507427:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763482.507434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763482.507448:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3462 00000100:00080000:1.0:1789763482.507462:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763482.507473:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763482.507520:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496239616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763482.507560:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763482.507566:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763482.507570:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763482.507577:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3462 00000100:00080000:1.0:1789763482.507585:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763482.507592:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763482.507606:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763482.507635:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496239744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763482.507635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496239616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763482.507665:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763482.507695:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763482.507699:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763482.507707:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3462 00000100:00080000:1.0:1789763482.507715:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763482.507722:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763482.507733:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789763482.507743:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763482.507760:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496239872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763482.507789:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763482.507795:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763482.507799:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763482.507809:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3462 00000100:00100000:3.0:1789763482.507810:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763482/real 1789763482] req@ffff8dd20504d180 x1876699496239616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763482.507817:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763482.507830:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763482.507845:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763482.507870:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496239616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763482.507872:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496240000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763482.507930:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763482.507942:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763482.507950:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496239616:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763482.507989:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496239744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763482.508034:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763482.508059:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763482/real 1789763482] req@ffff8dd20504e300 x1876699496239744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763482.508081:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496239744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763482.508096:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763482.508101:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763482.508107:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496239744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763482.508117:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496239872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763482.508151:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763482.508238:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763482/real 1789763482] req@ffff8dd20504d500 x1876699496239872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763482.508260:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496239872/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763482.508274:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763482.508279:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763482.508285:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496239872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763482.508294:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496240000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763482.508330:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763482.508349:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763482/real 1789763482] req@ffff8dd20504df80 x1876699496240000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763482.508569:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496240000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763482.508604:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763482.508611:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763482.508618:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496240000:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763483.083385:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763484.107360:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763485.131468:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763486.155385:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763487.179301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763487.627329:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763487.627349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763487.627356:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763487.627372:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3467 00000100:00080000:2.0:1789763487.627389:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763487.627401:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763487.627456:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496240128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763487.627504:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763487.627511:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763487.627515:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763487.627523:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3467 00000100:00080000:2.0:1789763487.627530:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763487.627537:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763487.627550:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763487.627571:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496240128:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763487.627576:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496240256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763487.627596:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763487.627601:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763487.627604:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763487.627608:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3467 00000100:00080000:2.0:1789763487.627614:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763487.627618:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763487.627625:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763487.627640:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496240384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763487.627655:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763487.627659:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763487.627661:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763487.627667:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3467 00000800:00000100:3.0:1789763487.627674:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763487.627677:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763487.627684:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763487.627691:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763487.627705:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496240512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763487.627732:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763487/real 1789763487] req@ffff8dd205264000 x1876699496240128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763487.627758:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496240128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763487.627771:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763487.627776:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763487.627782:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496240128:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763487.627810:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496240256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763487.627837:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763487.627855:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763487/real 1789763487] req@ffff8dd205267800 x1876699496240256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763487.627879:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496240256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763487.627895:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763487.627902:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763487.627909:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496240256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763487.627921:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496240384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763487.627960:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763487.627983:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763487/real 1789763487] req@ffff8dd205264e00 x1876699496240384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763487.628008:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496240384/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763487.628037:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763487.628044:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763487.628050:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496240384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763487.628062:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496240512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763487.628099:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763487.628119:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763487/real 1789763487] req@ffff8dd205264380 x1876699496240512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763487.628219:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496240512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763487.628239:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763487.628246:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763487.628254:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496240512:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763488.203400:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763489.227377:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763490.251325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763491.275356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763492.299375:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763492.747355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763492.747368:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763492.747375:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763492.747388:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3472 00000100:00080000:0.0:1789763492.747403:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763492.747413:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763492.747463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496240640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763492.747501:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763492.747506:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763492.747508:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763492.747513:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3472 00000100:00080000:0.0:1789763492.747517:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763492.747521:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763492.747534:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763492.747553:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496240768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763492.747568:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763492.747573:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763492.747576:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763492.747586:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3472 00000100:00100000:3.0:1789763492.747589:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496240640:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763492.747591:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763492.747605:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763492.747612:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763492.747632:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496240896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763492.747648:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763492.747654:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763492.747657:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763492.747661:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3472 00000100:00080000:0.0:1789763492.747667:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763492.747672:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763492.747678:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763492.747689:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763492.747697:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496241024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763492.747767:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763492/real 1789763492] req@ffff8dd211837100 x1876699496240640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763492.747793:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496240640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763492.747807:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763492.747812:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763492.747818:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496240640:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763492.747828:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496240768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763492.747857:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763492.747873:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763492/real 1789763492] req@ffff8dd211836300 x1876699496240768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763492.747888:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496240768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763492.747898:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763492.747902:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763492.747906:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496240768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763492.747936:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496240896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763492.747957:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763492.747969:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763492/real 1789763492] req@ffff8dd211837800 x1876699496240896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763492.747983:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496240896/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763492.747994:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763492.747998:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763492.748001:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496240896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763492.748008:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496241024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763492.748026:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763492.748037:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763492/real 1789763492] req@ffff8dd211837b80 x1876699496241024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763492.748052:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496241024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763492.748062:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763492.748065:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763492.748069:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496241024:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763493.323401:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763494.347461:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763495.371381:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763496.395444:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763497.419288:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763497.867270:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763497.867280:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763497.867285:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763497.867296:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3477 00000100:00080000:0.0:1789763497.867310:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763497.867335:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763497.867370:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496241152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763497.867410:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763497.867413:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763497.867415:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763497.867420:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3477 00000100:00080000:0.0:1789763497.867423:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763497.867427:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763497.867436:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763497.867468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496241280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763497.867478:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763497.867481:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763497.867483:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763497.867488:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3477 00000100:00080000:0.0:1789763497.867492:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763497.867498:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763497.867504:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763497.867513:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496241152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763497.867518:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496241408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763497.867528:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763497.867531:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763497.867532:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763497.867535:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3477 00000100:00080000:0.0:1789763497.867538:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763497.867541:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763497.867545:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763497.867554:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496241536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763497.867582:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763497.867626:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763497/real 1789763497] req@ffff8dd211837b80 x1876699496241152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763497.867646:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496241152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763497.867654:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763497.867657:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763497.867660:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496241152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763497.867666:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496241280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763497.867684:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763497.867693:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763497/real 1789763497] req@ffff8dd211837800 x1876699496241280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763497.867702:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496241280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763497.867712:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763497.867714:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763497.867724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496241280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763497.867743:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496241408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763497.867756:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763497.867763:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763497/real 1789763497] req@ffff8dd211836300 x1876699496241408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763497.867770:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496241408/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763497.867776:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763497.867778:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763497.867780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496241408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763497.867783:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496241536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763497.867793:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763497.867818:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763497/real 1789763497] req@ffff8dd211837100 x1876699496241536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763497.867833:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496241536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763497.867843:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763497.867846:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763497.867851:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496241536:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763498.443378:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763499.467325:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763500.491374:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763501.515359:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763502.539401:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763502.987352:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763502.987366:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763502.987372:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763502.987387:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3482 00000100:00080000:1.0:1789763502.987401:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763502.987410:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763502.987458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496241664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763502.987494:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763502.987499:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763502.987502:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763502.987507:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3482 00000100:00080000:1.0:1789763502.987511:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763502.987515:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763502.987527:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763502.987546:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496241792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763502.987562:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763502.987572:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789763502.987572:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496241664:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763502.987577:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763502.987590:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3482 00000100:00080000:1.0:1789763502.987596:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763502.987601:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763502.987608:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763502.987623:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496241920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763502.987638:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763502.987643:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763502.987645:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763502.987650:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3482 00000100:00080000:1.0:1789763502.987656:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763502.987661:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763502.987666:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763502.987673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763502.987687:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496242048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763502.987736:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763502/real 1789763502] req@ffff8dd20504e680 x1876699496241664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763502.987762:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496241664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763502.987775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763502.987781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763502.987786:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496241664:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763502.987815:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496241792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763502.987842:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763502.987856:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763502/real 1789763502] req@ffff8dd20504df80 x1876699496241792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763502.987871:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496241792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763502.987883:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763502.987887:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763502.987891:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496241792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763502.987898:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496241920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763502.987918:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763502.987930:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763502/real 1789763502] req@ffff8dd20504d500 x1876699496241920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763502.987945:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496241920/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763502.987955:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763502.987959:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763502.987962:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496241920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763502.987969:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496242048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763502.987988:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763502.987999:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763502/real 1789763502] req@ffff8dd20504e300 x1876699496242048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763502.988014:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496242048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763502.988050:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763502.988054:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763502.988058:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496242048:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763503.563367:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763504.587367:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763505.611393:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763506.635418:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763507.659398:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763508.107328:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763508.107359:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763508.107366:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763508.107382:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3487 00000100:00080000:0.0:1789763508.107395:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763508.107405:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763508.107455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496242176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763508.107519:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763508.107525:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763508.107528:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763508.107533:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3487 00000100:00080000:0.0:1789763508.107539:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763508.107545:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763508.107557:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763508.107581:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496242176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763508.107587:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496242304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763508.107601:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763508.107604:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763508.107606:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763508.107610:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3487 00000100:00080000:0.0:1789763508.107614:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763508.107619:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763508.107624:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763508.107636:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496242432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763508.107652:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789763508.107655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763508.107657:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763508.107661:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763508.107666:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3487 00000100:00080000:0.0:1789763508.107670:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763508.107674:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763508.107679:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763508.107690:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496242560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763508.107721:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763508/real 1789763508] req@ffff8dd211834700 x1876699496242176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763508.107753:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496242176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763508.107770:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763508.107775:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763508.107780:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496242176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763508.107812:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496242304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763508.107870:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763508.107894:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763508/real 1789763508] req@ffff8dd211835c00 x1876699496242304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763508.107919:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496242304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763508.107936:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763508.107942:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763508.107949:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496242304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763508.107957:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496242432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763508.108010:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763508.108047:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763508/real 1789763508] req@ffff8dd211836a00 x1876699496242432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763508.108063:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496242432/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763508.108075:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763508.108079:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763508.108083:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496242432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763508.108091:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496242560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763508.108119:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763508.108140:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763508/real 1789763508] req@ffff8dd211834000 x1876699496242560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763508.108180:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496242560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763508.108195:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763508.108265:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763508.108271:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496242560:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763508.683393:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763509.707337:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763510.731384:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763511.755412:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763512.779462:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763513.227391:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763513.227403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763513.227409:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763513.227422:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3492 00000100:00080000:1.0:1789763513.227434:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763513.227443:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763513.227499:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496242688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763513.227529:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763513.227537:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763513.227550:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763513.227555:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3492 00000100:00080000:1.0:1789763513.227559:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763513.227563:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763513.227575:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763513.227593:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496242816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763513.227609:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763513.227615:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763513.227620:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763513.227626:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496242688:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763513.227635:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3492 00000100:00080000:1.0:1789763513.227648:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763513.227656:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763513.227667:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763513.227719:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496242944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763513.227741:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763513.227749:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763513.227754:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763513.227766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763513.227767:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3492 00000100:00080000:1.0:1789763513.227784:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763513.227794:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763513.227808:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763513.227832:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763513/real 1789763513] req@ffff8dd20504d180 x1876699496242688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763513.227843:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496243072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763513.227874:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496242688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763513.227910:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763513.227916:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763513.227921:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496242688:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763513.227985:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496242816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763513.228028:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763513.228048:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763513/real 1789763513] req@ffff8dd20504ce00 x1876699496242816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763513.228065:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496242816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763513.228076:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763513.228080:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763513.228084:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496242816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763513.228091:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496242944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763513.228113:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763513.228125:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763513/real 1789763513] req@ffff8dd20504fb80 x1876699496242944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763513.228139:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496242944/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763513.228148:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763513.228151:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763513.228155:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496242944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763513.228161:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496243072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763513.228222:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763513.228236:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763513/real 1789763513] req@ffff8dd20504f800 x1876699496243072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763513.228250:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496243072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763513.228268:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763513.228272:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763513.228276:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496243072:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763513.803370:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763514.827403:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763515.851422:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763516.875411:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763517.899480:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763518.347258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763518.347275:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763518.347283:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763518.347299:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3497 00000100:00080000:1.0:1789763518.347317:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763518.347329:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763518.347387:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496243200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763518.347438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763518.347446:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763518.347450:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763518.347459:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3497 00000100:00080000:1.0:1789763518.347469:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763518.347479:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763518.347496:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763518.347521:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496243328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763518.347540:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763518.347546:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496243200:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763518.347548:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763518.347560:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763518.347577:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3497 00000100:00080000:1.0:1789763518.347583:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763518.347589:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763518.347597:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763518.347614:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496243456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763518.347627:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763518.347632:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763518.347635:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763518.347640:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3497 00000100:00080000:1.0:1789763518.347645:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763518.347649:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763518.347655:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763518.347684:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496243584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763518.347732:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763518.347804:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763518/real 1789763518] req@ffff8dd20504c700 x1876699496243200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763518.347833:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496243200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763518.347850:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763518.347856:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763518.347862:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496243200:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763518.347903:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496243328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763518.347975:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763518.348002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763518/real 1789763518] req@ffff8dd20504f100 x1876699496243328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763518.348031:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496243328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763518.348050:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763518.348058:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763518.348064:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496243328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763518.348075:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496243456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763518.348110:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763518.348126:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763518/real 1789763518] req@ffff8dd20504ed80 x1876699496243456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763518.348142:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496243456/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763518.348153:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763518.348157:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763518.348162:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496243456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763518.348217:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496243584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763518.348244:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763518.348259:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763518/real 1789763518] req@ffff8dd20504d880 x1876699496243584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763518.348274:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496243584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763518.348298:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763518.348303:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763518.348307:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496243584:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763518.923430:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763519.947375:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763520.971429:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763521.995439:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763523.019387:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763523.467360:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763523.467376:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763523.467384:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763523.467401:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3503 00000100:00080000:0.0:1789763523.467420:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763523.467432:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763523.467496:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496243712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763523.467537:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763523.467542:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763523.467546:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763523.467551:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3503 00000100:00080000:0.0:1789763523.467556:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763523.467560:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763523.467576:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763523.467596:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496243840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763523.467611:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763523.467620:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789763523.467622:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496243712:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763523.467624:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763523.467638:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3503 00000100:00080000:0.0:1789763523.467643:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763523.467647:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763523.467654:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763523.467669:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496243968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763523.467692:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763523.467697:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763523.467700:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763523.467704:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3503 00000100:00080000:0.0:1789763523.467708:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763523.467712:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763523.467718:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763523.467733:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496244096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763523.467749:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763523.467830:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763523/real 1789763523] req@ffff8dd211835f80 x1876699496243712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763523.467872:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496243712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763523.467920:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763523.467929:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763523.467938:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496243712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763523.467982:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496243840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763523.468035:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763523.468056:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763523/real 1789763523] req@ffff8dd211834380 x1876699496243840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763523.468074:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496243840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763523.468086:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763523.468090:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763523.468095:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496243840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763523.468102:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496243968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763523.468127:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763523.468139:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763523/real 1789763523] req@ffff8dd211836680 x1876699496243968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763523.468155:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496243968/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763523.468219:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763523.468223:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763523.468228:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496243968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763523.468236:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496244096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763523.468262:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763523.468275:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763523/real 1789763523] req@ffff8dd211834a80 x1876699496244096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763523.468290:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496244096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763523.468301:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763523.468305:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763523.468309:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496244096:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763524.043392:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763525.067442:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763526.091454:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763527.115361:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763528.139444:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763528.587320:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763528.587333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763528.587338:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763528.587351:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3508 00000100:00080000:3.0:1789763528.587364:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763528.587373:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763528.587422:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496244224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763528.587473:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763528.587479:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763528.587482:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763528.587487:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3508 00000100:00080000:3.0:1789763528.587492:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763528.587496:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763528.587508:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763528.587525:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496244352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763528.587539:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763528.587543:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763528.587545:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763528.587551:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3508 00000100:00080000:3.0:1789763528.587555:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763528.587560:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763528.587567:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763528.587592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496244480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763528.587608:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496244224:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763528.587611:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763528.587626:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763528.587632:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763528.587639:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3508 00000100:00080000:3.0:1789763528.587643:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763528.587648:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763528.587655:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763528.587670:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496244608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763528.587731:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763528.587789:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763528/real 1789763528] req@ffff8dd203144e00 x1876699496244224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763528.587813:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496244224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763528.587827:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763528.587832:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763528.587863:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496244224:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763528.587878:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496244352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763528.587911:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763528.587928:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763528/real 1789763528] req@ffff8dd203146a00 x1876699496244352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763528.587944:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496244352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763528.587955:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763528.587960:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763528.587964:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496244352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763528.587995:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496244480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763528.588017:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763528.588030:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763528/real 1789763528] req@ffff8dd203147b80 x1876699496244480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763528.588045:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496244480/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763528.588055:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763528.588058:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763528.588062:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496244480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763528.588069:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496244608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763528.588088:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763528.588099:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763528/real 1789763528] req@ffff8dd203146d80 x1876699496244608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763528.588113:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496244608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763528.588123:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763528.588126:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763528.588129:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496244608:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763529.163313:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763530.187463:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763531.211368:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763532.235473:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763533.260356:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763533.707329:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763533.707345:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763533.707353:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763533.707369:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3513 00000100:00080000:3.0:1789763533.707387:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763533.707400:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763533.707463:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496244736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763533.707526:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763533.707535:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763533.707538:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763533.707545:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3513 00000100:00080000:3.0:1789763533.707551:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763533.707557:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763533.707570:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763533.707592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496244864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763533.707609:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763533.707618:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789763533.707621:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496244736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763533.707622:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763533.707631:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3513 00000100:00080000:3.0:1789763533.707643:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763533.707649:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763533.707656:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763533.707676:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496244992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763533.707691:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763533.707696:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763533.707699:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763533.707706:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3513 00000100:00080000:3.0:1789763533.707711:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763533.707717:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763533.707724:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763533.707741:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763533.707743:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496245120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763533.707840:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763533/real 1789763533] req@ffff8dd203144380 x1876699496244736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763533.707880:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496244736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763533.707914:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763533.707923:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763533.707932:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496244736:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763533.707967:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496244864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763533.708003:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763533.708021:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763533/real 1789763533] req@ffff8dd203145500 x1876699496244864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763533.708043:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496244864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763533.708060:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763533.708067:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763533.708074:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496244864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763533.708086:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496244992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763533.708126:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763533.708148:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763533/real 1789763533] req@ffff8dd203144a80 x1876699496244992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763533.708163:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496244992/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763533.708218:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763533.708222:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763533.708227:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496244992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763533.708235:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496245120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763533.708266:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763533.708281:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763533/real 1789763533] req@ffff8dd203145f80 x1876699496245120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763533.708296:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496245120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763533.708306:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763533.708310:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763533.708329:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496245120:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763534.283305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763535.307455:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763536.331313:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763537.355512:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763538.379454:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763538.827334:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763538.827347:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763538.827352:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763538.827366:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3518 00000100:00080000:1.0:1789763538.827380:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763538.827390:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763538.827452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496245248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763538.827513:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763538.827524:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763538.827527:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763538.827534:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3518 00000100:00080000:1.0:1789763538.827541:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763538.827547:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763538.827562:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763538.827590:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496245376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763538.827614:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763538.827628:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496245248:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763538.827628:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763538.827636:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763538.827652:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3518 00000100:00080000:1.0:1789763538.827659:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763538.827668:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763538.827697:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763538.827723:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496245504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763538.827780:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763538.827787:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763538.827790:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763538.827796:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3518 00000100:00080000:1.0:1789763538.827804:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763538.827821:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763538.827837:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763538.827860:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763538.827887:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496245632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763538.827904:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763538/real 1789763538] req@ffff8dd20504c000 x1876699496245248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763538.827950:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496245248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763538.827972:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763538.827978:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763538.827984:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496245248:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763538.828024:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496245376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763538.828078:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763538.828110:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763538/real 1789763538] req@ffff8dd20504ca80 x1876699496245376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763538.828127:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496245376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763538.828139:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763538.828143:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763538.828147:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496245376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763538.828156:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496245504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763538.828224:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763538.828239:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763538/real 1789763538] req@ffff8dd20504ea00 x1876699496245504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763538.828255:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496245504/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763538.828266:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763538.828270:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763538.828274:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496245504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763538.828282:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496245632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763538.828304:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763538.828316:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763538/real 1789763538] req@ffff8dd20504c380 x1876699496245632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763538.828332:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496245632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763538.828342:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763538.828346:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763538.828349:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496245632:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763539.403314:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763540.427356:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763541.451380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763542.475349:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763543.499406:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763543.947350:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763543.947366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763543.947373:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763543.947389:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3523 00000100:00080000:0.0:1789763543.947406:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763543.947418:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763543.947473:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496245760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763543.947529:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763543.947537:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763543.947541:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763543.947553:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3523 00000100:00080000:0.0:1789763543.947562:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763543.947572:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763543.947589:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763543.947599:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496245760:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763543.947622:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496245888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763543.947646:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763543.947653:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763543.947657:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763543.947665:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3523 00000100:00080000:0.0:1789763543.947697:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763543.947707:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763543.947718:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763543.947775:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496246016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763543.947799:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763543.947810:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789763543.947812:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496245888:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1789763543.947819:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763543.947857:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3523 00000100:00080000:0.0:1789763543.947870:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763543.947879:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763543.947885:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496246016:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1789763543.947896:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763543.947934:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496246144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763543.947990:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496246144:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763544.523394:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763545.547422:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763546.571453:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763547.019646:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789763547.019732:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763547.019758:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763547.019815:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763547.019827:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763547.019840:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789763547.019880:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763543/real 1789763547] req@ffff8dd211834e00 x1876699496246144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763547.019933:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496246144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763547.019957:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763547.019966:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763547.019974:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496246144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763547.019989:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763543/real 1789763547] req@ffff8dd211835180 x1876699496245888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763547.020014:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496245888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763547.020031:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763547.020038:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763547.020045:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496245888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763547.020058:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763543/real 1789763547] req@ffff8dd211837480 x1876699496246016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763547.020082:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496246016/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763547.020099:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763547.020106:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763547.020113:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496246016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763547.020126:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763543/real 1789763547] req@ffff8dd211836d80 x1876699496245760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763547.020153:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496245760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763547.020209:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763547.020218:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763547.020225:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496245760:192.168.203.109@tcp:250:kworker.0 00010000:00010000:2.0:1789763547.595401:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763548.619366:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763549.067309:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763549.067319:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763549.067322:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763549.067332:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3528 00000100:00080000:3.0:1789763549.067359:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763549.067366:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763549.067397:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496246272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763549.067421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763549.067427:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763549.067429:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763549.067432:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3528 00000100:00080000:3.0:1789763549.067436:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763549.067440:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763549.067449:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763549.067463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496246400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763549.067467:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496246272:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763549.067479:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763549.067483:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763549.067485:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763549.067488:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3528 00000100:00080000:3.0:1789763549.067491:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763549.067494:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763549.067498:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763549.067538:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496246528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763549.067554:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763549.067582:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763549.067590:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763549.067596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763549.067620:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3528 00000100:00080000:3.0:1789763549.067629:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763549.067636:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763549.067646:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763549.067666:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496246656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763549.067711:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763549/real 1789763549] req@ffff8dd203147800 x1876699496246272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763549.067739:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496246272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763549.067754:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763549.067759:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763549.067772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496246272:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763549.067802:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496246400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763549.067837:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763549.067854:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763549/real 1789763549] req@ffff8dd203145c00 x1876699496246400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763549.067868:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496246400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763549.067890:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763549.067895:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763549.067899:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496246400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763549.067909:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496246528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763549.067940:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763549.067956:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763549/real 1789763549] req@ffff8dd203146300 x1876699496246528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763549.067971:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496246528/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763549.067981:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763549.067985:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763549.067988:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496246528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763549.067996:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496246656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763549.068020:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763549.068035:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763549/real 1789763549] req@ffff8dd203144700 x1876699496246656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763549.068050:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496246656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763549.068060:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763549.068064:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763549.068069:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496246656:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763549.643334:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763550.667423:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763551.691415:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763552.715483:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763553.739413:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763554.187284:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763554.187301:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763554.187321:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763554.187343:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3533 00000100:00080000:3.0:1789763554.187357:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763554.187369:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763554.187417:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496246784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763554.187452:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763554.187458:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763554.187465:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763554.187490:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3533 00000100:00080000:3.0:1789763554.187495:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763554.187500:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763554.187513:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763554.187531:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496246912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763554.187548:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763554.187552:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763554.187557:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763554.187563:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496246784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763554.187565:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3533 00000100:00080000:3.0:1789763554.187575:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763554.187586:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763554.187594:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763554.187613:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496247040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763554.187628:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763554.187633:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763554.187635:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763554.187641:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3533 00000100:00080000:3.0:1789763554.187646:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763554.187680:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763554.187687:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763554.187709:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763554.187719:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496247168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763554.187776:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763554/real 1789763554] req@ffff8dd203145180 x1876699496246784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763554.187802:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496246784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763554.187816:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763554.187822:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763554.187827:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496246784:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763554.187855:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496246912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763554.187889:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763554.187905:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763554/real 1789763554] req@ffff8dd203144000 x1876699496246912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763554.187921:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496246912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763554.187932:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763554.187936:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763554.187941:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496246912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763554.187964:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496247040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763554.187989:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763554.188002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763554/real 1789763554] req@ffff8dd203147100 x1876699496247040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763554.188017:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496247040/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763554.188028:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763554.188032:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763554.188036:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496247040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763554.188043:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496247168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763554.188062:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763554.188073:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763554/real 1789763554] req@ffff8dd203145880 x1876699496247168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763554.188087:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496247168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763554.188097:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763554.188101:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763554.188105:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496247168:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763554.763480:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763555.787456:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763556.811442:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763557.835317:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763558.859400:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763559.307315:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763559.307332:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763559.307339:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763559.307356:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3538 00000100:00080000:3.0:1789763559.307373:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763559.307382:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763559.307431:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496247296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763559.307471:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763559.307487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763559.307490:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763559.307495:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3538 00000100:00080000:3.0:1789763559.307500:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763559.307504:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763559.307517:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763559.307536:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496247424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763559.307552:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763559.307559:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789763559.307563:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496247296:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763559.307564:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763559.307580:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3538 00000100:00080000:3.0:1789763559.307584:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763559.307589:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763559.307596:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763559.307642:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496247552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763559.307663:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763559.307670:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763559.307675:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763559.307683:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3538 00000800:00000100:0.0:1789763559.307692:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763559.307693:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763559.307709:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763559.307723:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763559.307753:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763559/real 1789763559] req@ffff8dd203147480 x1876699496247296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763559.307755:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496247680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763559.307830:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496247296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763559.307848:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763559.307855:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763559.307860:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496247296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763559.307890:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496247424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763559.307917:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763559.307932:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763559/real 1789763559] req@ffff8dd203145880 x1876699496247424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763559.307947:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496247424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763559.307958:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763559.307962:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763559.307966:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496247424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763559.307973:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496247552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763559.307992:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763559.308003:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763559/real 1789763559] req@ffff8dd203147100 x1876699496247552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763559.308025:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496247552/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763559.308035:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763559.308039:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763559.308042:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496247552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763559.308049:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496247680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763559.308067:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763559.308078:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763559/real 1789763559] req@ffff8dd203144000 x1876699496247680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763559.308092:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496247680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763559.308101:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763559.308105:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763559.308108:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496247680:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763559.883349:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763560.907377:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763561.931372:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763562.955388:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763563.979386:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763564.427391:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763564.427404:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763564.427410:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763564.427423:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3544 00000100:00080000:2.0:1789763564.427437:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763564.427447:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763564.427496:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496247808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763564.427534:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763564.427539:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763564.427542:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763564.427547:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3544 00000100:00080000:2.0:1789763564.427551:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763564.427555:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763564.427567:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763564.427611:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496247936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763564.427626:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763564.427631:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763564.427634:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763564.427644:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3544 00000100:00100000:0.0:1789763564.427651:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496247808:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763564.427651:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763564.427662:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763564.427679:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763564.427693:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496248064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763564.427708:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763564.427712:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763564.427715:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763564.427720:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3544 00000100:00080000:2.0:1789763564.427727:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763564.427732:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763564.427738:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763564.427751:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496248192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763564.427771:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763564.427842:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763564/real 1789763564] req@ffff8dd205267100 x1876699496247808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763564.427895:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496247808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763564.427916:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763564.427925:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763564.427934:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496247808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763564.427950:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496247936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763564.427993:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763564.428018:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763564/real 1789763564] req@ffff8dd205264a80 x1876699496247936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763564.428043:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496247936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763564.428061:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763564.428068:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763564.428075:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496247936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763564.428115:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496248064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763564.428155:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763564.428232:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763564/real 1789763564] req@ffff8dd205267480 x1876699496248064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763564.428261:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496248064/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763564.428293:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763564.428300:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763564.428308:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496248064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763564.428320:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496248192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763564.428365:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763564.428389:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763564/real 1789763564] req@ffff8dd205266680 x1876699496248192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763564.428408:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496248192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763564.428419:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763564.428423:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763564.428427:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496248192:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763565.003449:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763566.027350:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763567.051451:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763568.075406:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763569.099368:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763569.547358:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763569.547370:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763569.547376:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763569.547388:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3549 00000100:00080000:0.0:1789763569.547402:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763569.547411:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763569.547462:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496248320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763569.547516:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763569.547521:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763569.547524:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763569.547530:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3549 00000100:00080000:0.0:1789763569.547557:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763569.547566:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763569.547583:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763569.547614:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496248448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763569.547618:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496248320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763569.547653:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763569.547661:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763569.547665:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763569.547677:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3549 00000100:00080000:0.0:1789763569.547700:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763569.547712:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763569.547716:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763569.547732:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763569.547768:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496248576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763569.547784:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763569/real 1789763569] req@ffff8dd211836d80 x1876699496248320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763569.547799:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763569.547812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763569.547816:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763569.547823:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496248320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763569.547833:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3549 00000100:00080000:1.0:1789763569.547852:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763569.547853:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763569.547865:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1789763569.547873:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763569.547879:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496248320:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789763569.547893:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763569.547924:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496248704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763569.547935:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496248448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763569.548005:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763569.548028:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763569/real 1789763569] req@ffff8dd211837480 x1876699496248448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763569.548045:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496248448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763569.548056:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763569.548061:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763569.548065:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496248448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763569.548073:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496248576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763569.548097:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763569.548109:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763569/real 1789763569] req@ffff8dd211835180 x1876699496248576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763569.548124:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496248576/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763569.548134:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763569.548138:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763569.548141:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496248576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763569.548159:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496248704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763569.548229:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763569.548245:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763569/real 1789763569] req@ffff8dd211834e00 x1876699496248704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763569.548261:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496248704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763569.548271:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763569.548275:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763569.548279:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496248704:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763570.123410:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763571.147368:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763572.171426:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763573.195505:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763574.219443:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763574.667337:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763574.667349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763574.667355:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763574.667368:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3554 00000100:00080000:0.0:1789763574.667381:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763574.667391:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763574.667453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496248832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763574.667490:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763574.667495:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763574.667525:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763574.667533:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3554 00000100:00080000:0.0:1789763574.667539:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763574.667545:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763574.667558:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763574.667578:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496248832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763574.667583:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496248960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763574.667613:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763574.667619:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763574.667624:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763574.667632:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3554 00000100:00080000:0.0:1789763574.667642:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763574.667651:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763574.667662:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763574.667684:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763574.667695:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496249088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763574.667721:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763574.667728:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763574.667732:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763574.667742:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763574/real 1789763574] req@ffff8dd211834e00 x1876699496248832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763574.667747:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3554 00000100:00080000:0.0:1789763574.667764:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763574.667778:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763574.667783:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496248832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763574.667797:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763574.667802:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763574.667811:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763574.667817:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496248832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763574.667837:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496249216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763574.667855:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496248960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763574.667912:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763574.667931:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763574/real 1789763574] req@ffff8dd211835180 x1876699496248960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763574.667948:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496248960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763574.667968:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763574.667973:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763574.667977:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496248960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763574.667985:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496249088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763574.668009:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763574.668021:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763574/real 1789763574] req@ffff8dd211837480 x1876699496249088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763574.668036:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496249088/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763574.668056:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763574.668060:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763574.668063:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496249088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763574.668080:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496249216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763574.668101:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763574.668113:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763574/real 1789763574] req@ffff8dd211836d80 x1876699496249216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763574.668127:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496249216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763574.668142:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763574.668149:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763574.668156:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496249216:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763575.243380:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763576.267446:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763577.291400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763578.315363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763579.339442:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763579.787328:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763579.787342:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763579.787348:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763579.787364:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3559 00000100:00080000:0.0:1789763579.787378:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763579.787388:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763579.787437:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496249344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763579.787497:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763579.787502:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763579.787504:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763579.787510:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3559 00000100:00080000:0.0:1789763579.787514:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763579.787519:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763579.787532:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763579.787551:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496249472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763579.787560:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496249344:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763579.787575:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763579.787582:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763579.787584:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763579.787590:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3559 00000100:00080000:0.0:1789763579.787594:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763579.787599:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763579.787606:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763579.787625:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496249600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763579.787640:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763579.787645:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763579.787647:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763579.787652:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3559 00000100:00080000:0.0:1789763579.787656:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763579.787660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763579.787668:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763579.787677:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763579.787695:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496249728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763579.787733:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763579/real 1789763579] req@ffff8dd211834a80 x1876699496249344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763579.787761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496249344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763579.787775:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763579.787781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763579.787786:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496249344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763579.787815:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496249472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763579.787843:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763579.787858:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763579/real 1789763579] req@ffff8dd211836680 x1876699496249472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763579.787874:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496249472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763579.787884:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763579.787897:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763579.787902:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496249472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763579.787909:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496249600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763579.787928:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763579.787941:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763579/real 1789763579] req@ffff8dd211834380 x1876699496249600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763579.787955:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496249600/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763579.787965:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763579.787969:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763579.787973:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496249600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763579.787979:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496249728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763579.787998:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763579.788009:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763579/real 1789763579] req@ffff8dd211835f80 x1876699496249728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763579.788023:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496249728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763579.788033:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763579.788036:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763579.788040:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496249728:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763580.363349:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763581.387416:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763582.411401:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763583.435405:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763584.459353:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763584.907316:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763584.907333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763584.907339:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763584.907352:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3564 00000100:00080000:0.0:1789763584.907366:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763584.907375:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763584.907450:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496249856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763584.907485:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763584.907490:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763584.907495:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763584.907501:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3564 00000100:00080000:0.0:1789763584.907509:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763584.907516:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763584.907529:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763584.907549:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496249984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763584.907563:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763584.907568:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763584.907570:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763584.907575:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3564 00000100:00080000:0.0:1789763584.907580:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763584.907585:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763584.907594:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763584.907598:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496249856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763584.907612:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496250112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763584.907630:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763584.907635:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763584.907637:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763584.907641:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3564 00000100:00080000:0.0:1789763584.907645:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763584.907650:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763584.907658:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763584.907676:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496250240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763584.907700:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763584.907757:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763584/real 1789763584] req@ffff8dd211834000 x1876699496249856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763584.907782:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496249856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763584.907796:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763584.907802:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763584.907808:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496249856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763584.907818:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496249984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763584.907846:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763584.907861:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763584/real 1789763584] req@ffff8dd211836a00 x1876699496249984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763584.907877:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496249984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763584.907887:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763584.907891:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763584.907896:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496249984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763584.907938:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496250112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763584.907962:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763584.907974:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763584/real 1789763584] req@ffff8dd211835c00 x1876699496250112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763584.907989:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496250112/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763584.908000:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763584.908004:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763584.908008:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496250112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763584.908015:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496250240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763584.908035:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763584.908046:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763584/real 1789763584] req@ffff8dd211834700 x1876699496250240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763584.908061:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496250240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763584.908071:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763584.908074:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763584.908078:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496250240:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763585.483465:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763586.507396:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763587.531386:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763588.555588:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763589.579448:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763590.027353:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763590.027366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763590.027372:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763590.027406:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3569 00000100:00080000:3.0:1789763590.027421:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763590.027433:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763590.027494:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496250368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763590.027531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763590.027536:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763590.027539:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763590.027544:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3569 00000100:00080000:3.0:1789763590.027548:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763590.027552:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763590.027563:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763590.027578:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496250496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763590.027601:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763590.027606:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763590.027609:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763590.027616:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3569 00000100:00080000:3.0:1789763590.027621:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763590.027627:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763590.027633:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763590.027646:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496250624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763590.027656:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496250368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763590.027666:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763590.027673:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763590.027676:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763590.027681:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3569 00000100:00080000:3.0:1789763590.027685:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763590.027690:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763590.027696:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763590.027713:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496250752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763590.027760:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763590.027816:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763590/real 1789763590] req@ffff8dd203145180 x1876699496250368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763590.027840:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496250368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763590.027854:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763590.027859:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763590.027864:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496250368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763590.027873:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496250496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763590.027898:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763590.027912:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763590/real 1789763590] req@ffff8dd203144700 x1876699496250496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763590.027928:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496250496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763590.027938:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763590.027942:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763590.027946:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496250496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763590.027975:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496250624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763590.027997:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763590.028009:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763590/real 1789763590] req@ffff8dd203146300 x1876699496250624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763590.028024:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496250624/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763590.028033:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763590.028037:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763590.028040:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496250624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763590.028047:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496250752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763590.028065:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763590.028076:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763590/real 1789763590] req@ffff8dd203145c00 x1876699496250752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763590.028090:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496250752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763590.028108:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763590.028111:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763590.028115:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496250752:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763590.603378:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763591.627363:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763592.651315:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763593.675459:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763594.699428:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763595.147325:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763595.147338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763595.147343:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763595.147387:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3574 00000100:00080000:3.0:1789763595.147403:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763595.147414:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763595.147469:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496250880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763595.147516:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763595.147528:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763595.147532:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763595.147541:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3574 00000100:00080000:3.0:1789763595.147551:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763595.147560:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763595.147578:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763595.147595:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496250880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763595.147613:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496251008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763595.147639:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763595.147647:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763595.147652:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763595.147661:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3574 00000100:00080000:3.0:1789763595.147670:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763595.147679:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763595.147692:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763595.147696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763595.147727:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496251136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763595.147744:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763595.147750:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763595.147753:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763595.147763:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3574 00000100:00100000:1.0:1789763595.147765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763595/real 1789763595] req@ffff8dd203147800 x1876699496250880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763595.147770:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763595.147785:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763595.147801:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763595.147810:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496250880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763595.147826:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496251264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763595.147834:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763595.147855:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763595.147867:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496250880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763595.147913:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496251008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763595.147957:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763595.147976:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763595/real 1789763595] req@ffff8dd203145f80 x1876699496251008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763595.147993:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496251008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763595.148005:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763595.148009:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763595.148013:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496251008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763595.148022:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496251136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763595.148056:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763595.148078:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763595/real 1789763595] req@ffff8dd203144a80 x1876699496251136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763595.148103:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496251136/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763595.148121:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763595.148128:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763595.148134:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496251136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763595.148143:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496251264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763595.148213:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763595.148230:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763595/real 1789763595] req@ffff8dd203145500 x1876699496251264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763595.148268:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496251264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763595.148279:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763595.148283:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763595.148287:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496251264:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763595.723402:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763596.747407:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763597.771385:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763598.795414:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763599.819438:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763600.267288:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763600.267306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763600.267336:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763600.267353:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3579 00000100:00080000:3.0:1789763600.267384:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763600.267394:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763600.267444:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496251392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763600.267496:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763600.267501:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763600.267504:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763600.267510:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3579 00000100:00080000:3.0:1789763600.267515:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763600.267520:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763600.267532:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763600.267553:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496251520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763600.267569:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763600.267576:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763600.267580:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496251392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763600.267581:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763600.267604:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3579 00000100:00080000:3.0:1789763600.267612:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763600.267621:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763600.267632:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763600.267671:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496251648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763600.267691:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789763600.267698:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763600.267715:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763600.267720:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763600.267729:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3579 00000100:00080000:3.0:1789763600.267734:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763600.267740:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763600.267748:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763600.267755:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763600/real 1789763600] req@ffff8dd203144380 x1876699496251392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763600.267770:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496251776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763600.267784:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496251392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763600.267818:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763600.267825:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763600.267832:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496251392:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763600.267867:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496251520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763600.267900:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763600.267915:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763600/real 1789763600] req@ffff8dd203146d80 x1876699496251520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763600.267931:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496251520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763600.267958:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763600.267962:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763600.267965:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496251520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763600.267970:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496251648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763600.267988:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763600.267997:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763600/real 1789763600] req@ffff8dd203147b80 x1876699496251648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763600.268009:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496251648/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763600.268023:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763600.268027:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763600.268030:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496251648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763600.268035:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496251776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763600.268059:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763600.268075:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763600/real 1789763600] req@ffff8dd203146a00 x1876699496251776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763600.268095:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496251776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763600.268125:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763600.268132:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763600.268139:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496251776:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763600.843409:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763601.867399:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763602.891455:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763603.915470:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763604.939444:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763605.387417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763605.387434:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763605.387440:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763605.387458:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3584 00000100:00080000:0.0:1789763605.387475:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763605.387487:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763605.387551:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496251904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763605.387594:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763605.387599:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763605.387602:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763605.387608:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3584 00000100:00080000:0.0:1789763605.387613:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763605.387618:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763605.387633:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763605.387653:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496252032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763605.387672:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763605.387676:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763605.387679:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763605.387688:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3584 00000100:00080000:0.0:1789763605.387694:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763605.387703:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496251904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763605.387706:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763605.387725:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763605.387739:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496252160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763605.387757:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763605.387762:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763605.387765:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763605.387771:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3584 00000100:00080000:0.0:1789763605.387777:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763605.387782:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763605.387788:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763605.387802:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496252288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763605.387824:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763605.387887:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763605/real 1789763605] req@ffff8dd211837100 x1876699496251904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763605.387923:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496251904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763605.387945:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763605.387954:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763605.387962:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496251904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763605.388006:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496252032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763605.388044:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763605.388062:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763605/real 1789763605] req@ffff8dd211836300 x1876699496252032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763605.388078:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496252032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763605.388090:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763605.388094:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763605.388098:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496252032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763605.388106:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496252160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763605.388127:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763605.388139:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763605/real 1789763605] req@ffff8dd211837800 x1876699496252160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763605.388154:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496252160/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763605.388165:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763605.388221:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763605.388225:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496252160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763605.388234:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496252288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763605.388259:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763605.388302:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763605/real 1789763605] req@ffff8dd211837b80 x1876699496252288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763605.388323:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496252288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763605.388340:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763605.388346:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763605.388352:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496252288:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763605.963421:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763606.987466:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763608.011508:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763609.035411:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763610.059432:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763610.507332:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763610.507348:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763610.507356:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763610.507373:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3590 00000100:00080000:0.0:1789763610.507391:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763610.507402:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763610.507470:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496252416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763610.507512:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763610.507518:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763610.507520:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763610.507526:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3590 00000100:00080000:0.0:1789763610.507530:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763610.507535:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763610.507549:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763610.507568:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496252544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763610.507582:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763610.507586:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763610.507591:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763610.507598:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3590 00000100:00080000:0.0:1789763610.507602:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763610.507613:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763610.507613:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496252416:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789763610.507645:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763610.507664:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496252672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763610.507683:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763610.507688:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763610.507690:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763610.507695:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3590 00000100:00080000:0.0:1789763610.507700:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763610.507707:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763610.507714:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763610.507728:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496252800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763610.507818:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496252544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763610.507891:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496252672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763610.507972:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496252800:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763611.083431:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763612.107442:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763613.131390:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763613.579609:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1789763613.579666:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763613.579692:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763613.579752:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763613.579769:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763613.579784:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789763613.579893:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763610/real 1789763613] req@ffff8dd211836300 x1876699496252672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.579941:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496252672/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.579987:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.579996:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763613.580004:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496252672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763613.580018:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763610/real 1789763613] req@ffff8dd211837100 x1876699496252800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.580041:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496252800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.580056:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.580062:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763613.580069:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496252800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763613.580080:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763610/real 1789763613] req@ffff8dd211837b80 x1876699496252416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.580103:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496252416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.580128:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.580135:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763613.580142:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496252416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763613.580155:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763610/real 1789763613] req@ffff8dd211837800 x1876699496252544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.580260:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496252544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.580278:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.580284:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763613.580290:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496252544:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763614.155407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763615.179434:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763615.627305:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763615.627323:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763615.627329:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763615.627347:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3595 00000100:00080000:0.0:1789763615.627366:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763615.627380:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763615.627444:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496252928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763615.627486:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763615.627492:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763615.627494:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763615.627500:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3595 00000100:00080000:0.0:1789763615.627505:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763615.627510:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763615.627525:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763615.627542:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496253056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763615.627560:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763615.627567:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763615.627573:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763615.627583:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3595 00000100:00080000:0.0:1789763615.627591:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763615.627609:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496252928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763615.627612:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763615.627640:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763615.627673:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496253184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763615.627702:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763615.627709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763615.627715:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763615.627725:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3595 00000100:00080000:0.0:1789763615.627759:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763615.627775:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763615.627778:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763615.627799:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763615.627838:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496253312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763615.627873:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763615/real 1789763615] req@ffff8dd211834700 x1876699496252928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763615.627917:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496252928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763615.627941:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763615.627950:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763615.627958:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496252928:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763615.627976:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496253056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763615.628030:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763615.628071:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763615/real 1789763615] req@ffff8dd211835c00 x1876699496253056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763615.628099:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496253056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763615.628116:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763615.628123:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763615.628130:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496253056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763615.628272:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496253184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763615.628323:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763615.628351:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763615/real 1789763615] req@ffff8dd211836a00 x1876699496253184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763615.628393:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496253184/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763615.628419:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763615.628427:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763615.628434:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496253184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763615.628449:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496253312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763615.628507:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763615.628534:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763615/real 1789763615] req@ffff8dd211834000 x1876699496253312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763615.628562:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496253312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763615.628579:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763615.628588:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763615.628595:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496253312:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763616.203413:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763617.227507:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763618.251412:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763619.275398:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763620.299392:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763620.747354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763620.747371:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763620.747378:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763620.747393:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3600 00000100:00080000:1.0:1789763620.747409:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763620.747423:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763620.747478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496253440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763620.747531:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763620.747541:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763620.747545:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763620.747553:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3600 00000100:00080000:1.0:1789763620.747561:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763620.747569:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763620.747585:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763620.747615:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496253568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763620.747644:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496253440:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763620.747644:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763620.747661:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763620.747668:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763620.747677:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3600 00000100:00080000:1.0:1789763620.747685:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763620.747694:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763620.747705:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763620.747730:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496253696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763620.747751:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789763620.747762:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763620.747765:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763620.747773:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763620.747784:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3600 00000100:00080000:1.0:1789763620.747794:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763620.747803:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763620.747816:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763620.747838:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496253824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763620.747855:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763620/real 1789763620] req@ffff8dd20504dc00 x1876699496253440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763620.747889:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496253440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763620.747913:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763620.747919:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763620.747924:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496253440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763620.747954:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496253568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763620.747986:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763620.748002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763620/real 1789763620] req@ffff8dd20504c380 x1876699496253568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763620.748028:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496253568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763620.748038:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763620.748042:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763620.748046:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496253568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763620.748053:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496253696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763620.748074:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763620.748086:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763620/real 1789763620] req@ffff8dd20504ea00 x1876699496253696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763620.748100:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496253696/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763620.748110:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763620.748114:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763620.748117:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496253696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763620.748124:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496253824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763620.748142:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763620.748153:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763620/real 1789763620] req@ffff8dd20504ca80 x1876699496253824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763620.748236:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496253824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763620.748255:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763620.748259:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763620.748263:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496253824:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763621.323373:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763622.347400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763623.371359:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763624.395396:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763625.419380:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763625.867322:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763625.867338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763625.867344:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763625.867360:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3605 00000100:00080000:0.0:1789763625.867375:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763625.867386:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763625.867440:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496253952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763625.867493:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763625.867501:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763625.867505:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763625.867514:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3605 00000100:00080000:0.0:1789763625.867523:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763625.867532:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763625.867547:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763625.867550:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496253952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763625.867575:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496254080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763625.867599:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763625.867606:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763625.867610:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763625.867617:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3605 00000100:00080000:0.0:1789763625.867625:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763625.867632:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763625.867641:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763625.867646:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763625.867673:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496254208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763625.867695:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763625.867703:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763625/real 1789763625] req@ffff8dd211835f80 x1876699496253952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763625.867707:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763625.867716:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763625.867735:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3605 00000100:00080000:0.0:1789763625.867745:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763625.867753:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496253952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763625.867756:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763625.867775:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763625.867781:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763625.867791:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763625.867799:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496253952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763625.867807:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496254336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763625.867839:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496254080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763625.867883:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763625.867909:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763625/real 1789763625] req@ffff8dd211834380 x1876699496254080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763625.867935:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496254080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763625.867952:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763625.867959:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763625.867966:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496254080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763625.867979:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496254208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763625.868025:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763625.868049:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763625/real 1789763625] req@ffff8dd211836680 x1876699496254208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763625.868075:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496254208/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763625.868087:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763625.868092:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763625.868096:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496254208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763625.868104:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496254336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763625.868163:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763625.868214:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763625/real 1789763625] req@ffff8dd211834a80 x1876699496254336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763625.868230:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496254336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763625.868244:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763625.868248:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763625.868253:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496254336:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763626.443362:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763627.467531:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763628.491437:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763629.515439:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763630.539368:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763630.987338:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763630.987351:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763630.987358:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763630.987370:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3610 00000100:00080000:1.0:1789763630.987385:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763630.987394:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763630.987449:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496254464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763630.987500:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763630.987508:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763630.987512:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763630.987521:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3610 00000100:00080000:1.0:1789763630.987530:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763630.987540:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763630.987556:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763630.987562:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496254464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763630.987589:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496254592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763630.987616:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763630.987620:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763630.987624:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763630.987630:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3610 00000100:00080000:1.0:1789763630.987650:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763630.987656:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1789763630.987665:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1789763630.987666:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763630.987705:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496254720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763630.987721:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763630.987729:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789763630.987730:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763630/real 1789763630] req@ffff8dd20504c000 x1876699496254464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763630.987734:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763630.987749:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3610 00000100:00080000:1.0:1789763630.987759:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763630.987770:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763630.987790:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763630.987801:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496254464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763630.987814:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496254848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763630.987823:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763630.987837:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763630.987846:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496254464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763630.987881:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496254592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763630.987910:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763630.987926:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763630/real 1789763630] req@ffff8dd20504d880 x1876699496254592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763630.987941:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496254592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763630.987952:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763630.987956:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763630.987960:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496254592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763630.987967:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496254720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763630.987987:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763630.987998:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763630/real 1789763630] req@ffff8dd20504ed80 x1876699496254720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763630.988012:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496254720/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763630.988031:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763630.988035:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763630.988038:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496254720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763630.988045:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496254848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763630.988063:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763630.988074:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763630/real 1789763630] req@ffff8dd20504f100 x1876699496254848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763630.988118:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496254848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763630.988130:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763630.988134:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763630.988138:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496254848:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763631.563365:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763632.587382:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763633.611359:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763634.635366:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763635.659395:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763636.107340:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763636.107354:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763636.107359:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763636.107372:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3615 00000100:00080000:3.0:1789763636.107387:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763636.107396:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763636.107443:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144e00 x1876699496254976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763636.107481:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763636.107489:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763636.107492:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763636.107497:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3615 00000100:00080000:3.0:1789763636.107501:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763636.107505:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763636.107518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763636.107550:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146a00 x1876699496255104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763636.107560:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496254976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763636.107577:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763636.107583:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763636.107585:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763636.107592:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3615 00000100:00080000:3.0:1789763636.107598:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763636.107603:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763636.107610:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763636.107629:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147b80 x1876699496255232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763636.107644:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789763636.107658:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763636.107664:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763636.107669:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763636.107678:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3615 00000100:00080000:3.0:1789763636.107685:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763636.107690:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763636.107699:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763636.107716:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146d80 x1876699496255360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763636.107725:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763636/real 1789763636] req@ffff8dd203144e00 x1876699496254976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763636.107763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144e00 x1876699496254976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763636.107787:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763636.107793:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763636.107799:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496254976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763636.107827:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496255104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763636.107855:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763636.107871:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763636/real 1789763636] req@ffff8dd203146a00 x1876699496255104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763636.107887:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146a00 x1876699496255104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763636.107897:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763636.107901:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763636.107905:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496255104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763636.107912:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496255232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763636.107931:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763636.107943:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763636/real 1789763636] req@ffff8dd203147b80 x1876699496255232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763636.107956:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147b80 x1876699496255232/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763636.107984:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763636.107988:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763636.107992:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496255232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763636.108000:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496255360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763636.108022:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763636.108035:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763636/real 1789763636] req@ffff8dd203146d80 x1876699496255360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763636.108076:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146d80 x1876699496255360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763636.108087:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763636.108091:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763636.108095:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496255360:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763636.683375:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763637.707409:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763638.731420:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763639.755410:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763640.779402:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763641.227342:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763641.227359:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763641.227366:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763641.227383:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3620 00000100:00080000:0.0:1789763641.227401:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763641.227412:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763641.227493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496255488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763641.227545:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763641.227553:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763641.227556:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763641.227567:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3620 00000100:00080000:0.0:1789763641.227576:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763641.227585:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763641.227605:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763641.227625:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496255488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763641.227641:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496255616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763641.227661:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763641.227667:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763641.227670:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763641.227678:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3620 00000100:00080000:0.0:1789763641.227684:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763641.227690:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763641.227698:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763641.227717:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496255744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763641.227731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763641.227736:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763641.227738:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763641.227743:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3620 00000100:00080000:0.0:1789763641.227748:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763641.227756:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763641.227763:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763641.227766:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763641.227786:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496255872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763641.227845:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763641/real 1789763641] req@ffff8dd211836d80 x1876699496255488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763641.227874:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496255488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763641.227889:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763641.227895:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763641.227901:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496255488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763641.227937:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496255616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763641.227971:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763641.227987:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763641/real 1789763641] req@ffff8dd211837480 x1876699496255616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763641.228003:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496255616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763641.228060:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763641.228065:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763641.228070:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496255616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763641.228080:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496255744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763641.228110:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763641.228126:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763641/real 1789763641] req@ffff8dd211835180 x1876699496255744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763641.228140:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496255744/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763641.228151:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763641.228155:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763641.228159:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496255744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763641.228210:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496255872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763641.228237:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763641.228252:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763641/real 1789763641] req@ffff8dd211834e00 x1876699496255872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763641.228267:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496255872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763641.228277:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763641.228281:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763641.228286:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496255872:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763641.803406:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763642.827384:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763643.851413:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763644.875395:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763645.899367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763646.347299:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763646.347312:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763646.347318:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763646.347330:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3625 00000100:00080000:0.0:1789763646.347344:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763646.347353:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763646.347402:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496256000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763646.347440:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763646.347446:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763646.347448:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763646.347454:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3625 00000100:00080000:0.0:1789763646.347458:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763646.347462:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763646.347475:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763646.347490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496256128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763646.347503:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763646.347507:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763646.347511:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763646.347516:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3625 00000100:00080000:0.0:1789763646.347522:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763646.347526:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763646.347532:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763646.347544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496256256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763646.347554:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496256000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763646.347565:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763646.347571:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763646.347574:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763646.347583:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3625 00000100:00080000:0.0:1789763646.347590:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763646.347598:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763646.347608:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763646.347636:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496256384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763646.347657:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763646.347728:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763646/real 1789763646] req@ffff8dd211834e00 x1876699496256000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763646.347755:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496256000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763646.347770:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763646.347775:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763646.347781:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496256000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763646.347791:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496256128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763646.347823:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763646.347839:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763646/real 1789763646] req@ffff8dd211835180 x1876699496256128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763646.347854:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496256128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763646.347865:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763646.347868:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763646.347872:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496256128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763646.347904:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496256256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763646.347926:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763646.347940:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763646/real 1789763646] req@ffff8dd211837480 x1876699496256256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763646.347954:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496256256/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763646.347964:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763646.347968:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763646.348004:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496256256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763646.348014:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496256384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763646.348042:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763646.348057:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763646/real 1789763646] req@ffff8dd211836d80 x1876699496256384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763646.348072:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496256384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763646.348083:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763646.348087:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763646.348091:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496256384:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763646.923324:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763647.947400:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763648.971400:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763649.995415:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763651.019484:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763651.467337:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763651.467349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763651.467355:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763651.467368:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3631 00000100:00080000:1.0:1789763651.467383:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763651.467392:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763651.467441:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496256512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763651.467491:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763651.467497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763651.467499:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763651.467505:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3631 00000100:00080000:1.0:1789763651.467509:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763651.467514:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763651.467526:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763651.467542:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496256640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763651.467556:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763651.467563:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763651.467568:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763651.467569:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496256512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763651.467576:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3631 00000100:00080000:1.0:1789763651.467589:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763651.467593:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763651.467600:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763651.467617:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496256768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763651.467631:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763651.467636:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763651.467639:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763651.467645:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3631 00000100:00080000:1.0:1789763651.467661:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763651.467671:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763651.467672:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763651.467685:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763651.467703:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496256896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763651.467756:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763651/real 1789763651] req@ffff8dd20504c700 x1876699496256512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763651.467783:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496256512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763651.467807:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763651.467813:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763651.467818:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496256512:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763651.467846:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496256640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763651.467873:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763651.467887:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763651/real 1789763651] req@ffff8dd20504f800 x1876699496256640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763651.467903:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496256640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763651.467919:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763651.467923:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763651.467927:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496256640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763651.467965:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496256768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763651.467997:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763651.468011:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763651/real 1789763651] req@ffff8dd20504fb80 x1876699496256768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763651.468026:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496256768/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763651.468037:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763651.468041:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763651.468045:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496256768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763651.468051:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496256896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763651.468074:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763651.468085:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763651/real 1789763651] req@ffff8dd20504ce00 x1876699496256896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763651.468109:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496256896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763651.468119:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763651.468122:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763651.468126:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496256896:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763652.043437:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763653.067577:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763654.091391:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763655.115349:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763656.139505:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763656.587404:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763656.587418:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763656.587423:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763656.587436:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3636 00000100:00080000:0.0:1789763656.587451:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763656.587474:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763656.587544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496257024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763656.587601:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763656.587607:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763656.587610:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763656.587617:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3636 00000100:00080000:0.0:1789763656.587625:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763656.587632:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763656.587646:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763656.587667:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496257024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763656.587669:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496257152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763656.587718:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763656.587723:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763656.587726:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763656.587734:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3636 00000100:00080000:0.0:1789763656.587738:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763656.587743:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763656.587748:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763656.587765:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496257280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763656.587786:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763656.587791:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763656.587795:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763656.587800:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3636 00000100:00080000:0.0:1789763656.587832:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763656.587837:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763656.587854:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763656.587868:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763656.587871:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496257408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763656.587975:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763656/real 1789763656] req@ffff8dd211834a80 x1876699496257024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763656.588020:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496257024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763656.588043:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763656.588052:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763656.588060:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496257024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763656.588110:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496257152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763656.588155:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763656.588175:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763656/real 1789763656] req@ffff8dd211836680 x1876699496257152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763656.588237:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496257152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763656.588246:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763656.588250:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763656.588253:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496257152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763656.588277:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496257280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763656.588315:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763656.588339:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763656/real 1789763656] req@ffff8dd211834380 x1876699496257280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763656.588363:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496257280/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763656.588380:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763656.588388:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763656.588393:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496257280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763656.588420:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496257408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763656.588442:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763656.588456:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763656/real 1789763656] req@ffff8dd211835f80 x1876699496257408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763656.588490:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496257408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763656.588506:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763656.588512:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763656.588519:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496257408:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763657.163475:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763658.187422:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763659.211431:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763660.235425:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763661.259448:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763661.707282:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763661.707292:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763661.707298:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763661.707309:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3641 00000100:00080000:0.0:1789763661.707322:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763661.707330:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763661.707371:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496257536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763661.707403:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763661.707407:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763661.707409:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763661.707414:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3641 00000100:00080000:0.0:1789763661.707418:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763661.707422:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763661.707432:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763661.707447:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496257664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763661.707461:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763661.707465:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763661.707468:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763661.707473:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496257536:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763661.707477:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3641 00000100:00080000:0.0:1789763661.707487:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763661.707493:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763661.707500:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763661.707523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496257792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763661.707541:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763661.707547:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763661.707551:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763661.707557:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3641 00000100:00080000:0.0:1789763661.707565:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763661.707573:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763661.707579:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763661.707588:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763661.707635:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496257920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763661.707676:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763661/real 1789763661] req@ffff8dd211834000 x1876699496257536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763661.707705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496257536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763661.707721:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763661.707726:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763661.707731:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496257536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763661.707763:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496257664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763661.707809:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763661.707825:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763661/real 1789763661] req@ffff8dd211836a00 x1876699496257664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763661.707838:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496257664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763661.707847:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763661.707851:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763661.707855:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496257664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763661.707884:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496257792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763661.707929:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763661.707949:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763661/real 1789763661] req@ffff8dd211835c00 x1876699496257792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763661.707994:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496257792/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763661.708006:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763661.708010:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763661.708014:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496257792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763661.708021:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496257920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763661.708046:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763661.708059:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763661/real 1789763661] req@ffff8dd211834700 x1876699496257920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763661.708071:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496257920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763661.708080:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763661.708084:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763661.708088:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496257920:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763662.283293:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763663.307450:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763664.331447:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763665.355387:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763666.379525:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763666.827311:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763666.827324:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763666.827330:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763666.827342:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3646 00000100:00080000:0.0:1789763666.827357:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763666.827370:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763666.827427:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496258048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763666.827461:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763666.827466:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763666.827469:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763666.827475:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3646 00000100:00080000:0.0:1789763666.827480:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763666.827486:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763666.827499:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763666.827518:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496258048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763666.827521:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496258176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763666.827548:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763666.827552:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763666.827555:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763666.827560:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3646 00000100:00080000:0.0:1789763666.827565:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763666.827570:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763666.827577:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763666.827592:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496258304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763666.827605:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763666.827609:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763666.827612:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763666.827621:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763666.827630:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3646 00000100:00080000:0.0:1789763666.827642:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763666.827652:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763666.827663:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763666.827685:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763666/real 1789763666] req@ffff8dd211837800 x1876699496258048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763666.827695:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496258432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763666.827767:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496258048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763666.827801:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763666.827807:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763666.827812:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496258048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763666.827871:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496258176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763666.827906:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763666.827924:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763666/real 1789763666] req@ffff8dd211837b80 x1876699496258176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763666.827940:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496258176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763666.827951:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763666.827956:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763666.827960:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496258176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763666.827968:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496258304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763666.827990:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763666.828002:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763666/real 1789763666] req@ffff8dd211837100 x1876699496258304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763666.828017:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496258304/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763666.828027:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763666.828031:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763666.828034:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496258304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763666.828042:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496258432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763666.828061:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763666.828073:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763666/real 1789763666] req@ffff8dd211836300 x1876699496258432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763666.828088:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496258432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763666.828110:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763666.828115:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763666.828121:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496258432:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763667.403408:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763668.427431:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763669.451431:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763670.475442:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763671.499407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763671.947370:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763671.947384:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763671.947391:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763671.947406:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3651 00000100:00080000:1.0:1789763671.947424:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763671.947436:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763671.947496:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496258560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763671.947558:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763671.947563:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763671.947566:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763671.947572:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3651 00000100:00080000:1.0:1789763671.947578:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763671.947585:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763671.947597:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763671.947618:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496258688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763671.947632:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763671.947637:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763671.947639:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763671.947645:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3651 00000100:00080000:1.0:1789763671.947649:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763671.947657:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496258560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763671.947659:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763671.947674:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763671.947692:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496258816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763671.947707:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763671.947712:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763671.947715:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763671.947720:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3651 00000100:00080000:1.0:1789763671.947744:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763671.947749:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763671.947756:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763671.947766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763671.947775:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496258944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763671.947876:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763671/real 1789763671] req@ffff8dd20504d180 x1876699496258560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763671.947903:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496258560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763671.947918:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763671.947923:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763671.947929:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496258560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763671.947939:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496258688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763671.947971:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763671.947987:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763671/real 1789763671] req@ffff8dd20504e300 x1876699496258688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763671.948003:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496258688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763671.948013:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763671.948018:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763671.948022:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496258688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763671.948055:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496258816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763671.948077:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763671.948090:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763671/real 1789763671] req@ffff8dd20504d500 x1876699496258816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763671.948104:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496258816/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763671.948115:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763671.948118:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763671.948122:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496258816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763671.948129:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496258944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763671.948148:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763671.948159:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763671/real 1789763671] req@ffff8dd20504df80 x1876699496258944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763671.948211:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496258944/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763671.948221:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763671.948225:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763671.948229:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496258944:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763672.523459:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763673.547379:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763674.571363:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763675.595458:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763676.619391:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763677.067416:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763677.067434:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763677.067443:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763677.067460:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3656 00000100:00080000:2.0:1789763677.067479:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763677.067491:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763677.067557:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496259072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763677.067599:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763677.067605:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763677.067607:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763677.067613:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3656 00000100:00080000:2.0:1789763677.067617:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763677.067622:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763677.067637:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763677.067660:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496259200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763677.067675:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763677.067680:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763677.067684:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763677.067691:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3656 00000100:00080000:2.0:1789763677.067695:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763677.067708:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496259072:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763677.067708:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763677.067723:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763677.067788:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496259328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763677.067806:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763677.067811:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763677.067814:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763677.067819:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3656 00000100:00080000:2.0:1789763677.067825:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763677.067832:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763677.067841:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763677.067862:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496259456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763677.067916:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496259200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763677.067972:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496259328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763677.068004:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496259456:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763677.643422:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763678.667529:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763679.691432:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763680.139526:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1789763680.139580:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763680.139601:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763680.139648:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763680.139665:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1789763680.139680:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789763680.139788:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763677/real 1789763680] req@ffff8dd205267480 x1876699496259328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763680.139822:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496259328/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763680.139841:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763680.139850:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763680.139859:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496259328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763680.139874:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763677/real 1789763680] req@ffff8dd205264a80 x1876699496259456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763680.139899:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496259456/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763680.139915:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763680.139922:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763680.139929:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496259456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763680.139941:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763677/real 1789763680] req@ffff8dd205265500 x1876699496259072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763680.139979:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496259072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763680.139995:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763680.140001:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763680.140007:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496259072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763680.140018:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763677/real 1789763680] req@ffff8dd205266680 x1876699496259200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763680.140039:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496259200/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763680.140053:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763680.140058:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763680.140064:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496259200:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763680.715383:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763681.739436:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763682.187388:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763682.187402:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763682.187407:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763682.187420:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3661 00000100:00080000:1.0:1789763682.187434:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763682.187442:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763682.187491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496259584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763682.187550:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763682.187566:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763682.187571:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763682.187581:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3661 00000100:00080000:1.0:1789763682.187590:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763682.187602:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496259584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763682.187603:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763682.187621:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763682.187649:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496259712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763682.187683:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763682.187692:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763682.187697:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763682.187732:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3661 00000100:00080000:1.0:1789763682.187756:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763682.187771:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763682.187787:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763682.187802:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763682.187834:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496259840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763682.187852:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763682/real 1789763682] req@ffff8dd20504e680 x1876699496259584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763682.187865:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763682.187881:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763682.187898:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763682.187908:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496259584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763682.187916:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3661 00000100:00080000:1.0:1789763682.187931:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763682.187945:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763682.187946:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1789763682.187966:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1789763682.187968:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763682.187981:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496259584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763682.188004:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496259968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763682.188032:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496259712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763682.188106:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763682.188135:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763682/real 1789763682] req@ffff8dd20504df80 x1876699496259712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763682.188163:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496259712/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763682.188219:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763682.188227:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763682.188235:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496259712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763682.188248:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496259840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763682.188293:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763682.188321:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763682/real 1789763682] req@ffff8dd20504d500 x1876699496259840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763682.188358:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496259840/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763682.188376:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763682.188383:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763682.188390:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496259840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763682.188428:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496259968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763682.188474:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763682.188499:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763682/real 1789763682] req@ffff8dd20504e300 x1876699496259968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763682.188524:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496259968/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763682.188542:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763682.188548:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763682.188554:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496259968:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763682.763435:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763683.787507:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763684.811461:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763685.835402:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763686.859367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763687.307451:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763687.307467:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763687.307474:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763687.307491:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3666 00000100:00080000:2.0:1789763687.307511:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763687.307525:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763687.307593:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496260096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763687.307629:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763687.307634:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763687.307637:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763687.307643:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3666 00000100:00080000:2.0:1789763687.307649:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763687.307654:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763687.307667:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763687.307728:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496260096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763687.307740:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496260224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763687.307766:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763687.307772:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763687.307775:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763687.307781:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3666 00000100:00080000:2.0:1789763687.307788:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763687.307794:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763687.307804:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763687.307821:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496260352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763687.307838:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763687.307842:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763687.307850:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763687.307854:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763687.307861:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3666 00000100:00080000:2.0:1789763687.307867:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763687.307872:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763687.307878:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763687.307895:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496260480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763687.307905:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763687/real 1789763687] req@ffff8dd205267100 x1876699496260096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763687.307947:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496260096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763687.307970:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763687.307979:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763687.307987:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496260096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763687.308025:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496260224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763687.308062:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763687.308080:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763687/real 1789763687] req@ffff8dd205264380 x1876699496260224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763687.308097:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496260224/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763687.308108:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763687.308113:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763687.308117:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496260224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763687.308124:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496260352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763687.308147:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763687.308159:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763687/real 1789763687] req@ffff8dd205264e00 x1876699496260352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763687.308208:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496260352/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763687.308220:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763687.308224:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763687.308229:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496260352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763687.308236:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496260480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763687.308261:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763687.308273:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763687/real 1789763687] req@ffff8dd205267800 x1876699496260480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763687.308288:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496260480/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763687.308306:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763687.308310:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763687.308314:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496260480:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763687.883420:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763688.907405:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763689.931399:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763690.955373:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763691.979430:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763692.427351:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763692.427363:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763692.427369:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763692.427381:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3672 00000100:00080000:0.0:1789763692.427395:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763692.427404:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763692.427452:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496260608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763692.427492:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763692.427498:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763692.427500:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763692.427505:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3672 00000100:00080000:0.0:1789763692.427510:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763692.427514:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763692.427526:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763692.427542:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496260736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763692.427557:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763692.427564:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763692.427564:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496260608:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763692.427571:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763692.427586:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3672 00000100:00080000:0.0:1789763692.427590:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763692.427594:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763692.427601:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763692.427616:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496260864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763692.427657:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763692.427662:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763692.427665:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763692.427670:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3672 00000100:00080000:0.0:1789763692.427677:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763692.427681:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763692.427687:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763692.427696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763692.427708:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496260992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763692.427760:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763692/real 1789763692] req@ffff8dd211836300 x1876699496260608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763692.427787:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496260608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763692.427801:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763692.427808:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763692.427813:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496260608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763692.427845:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496260736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763692.427876:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763692.427892:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763692/real 1789763692] req@ffff8dd211837100 x1876699496260736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763692.427910:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496260736/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763692.427928:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763692.427935:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763692.427941:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496260736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763692.427954:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496260864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763692.428012:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763692.428036:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763692/real 1789763692] req@ffff8dd211837b80 x1876699496260864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763692.428053:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496260864/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763692.428064:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763692.428068:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763692.428072:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496260864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763692.428080:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496260992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763692.428107:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763692.428120:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763692/real 1789763692] req@ffff8dd211837800 x1876699496260992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763692.428134:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496260992/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763692.428144:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763692.428147:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763692.428151:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496260992:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763693.003414:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763694.027449:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763695.051482:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763696.075407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763697.099441:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763697.547321:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763697.547333:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763697.547339:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763697.547352:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3677 00000100:00080000:0.0:1789763697.547367:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763697.547377:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763697.547424:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496261120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763697.547461:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763697.547466:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763697.547468:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763697.547473:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3677 00000100:00080000:0.0:1789763697.547478:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763697.547484:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763697.547497:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763697.547514:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496261248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763697.547530:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763697.547534:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496261120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763697.547537:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763697.547544:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763697.547556:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3677 00000100:00080000:0.0:1789763697.547560:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763697.547565:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763697.547572:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763697.547589:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496261376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763697.547637:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763697.547642:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763697.547645:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763697.547651:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3677 00000100:00080000:0.0:1789763697.547657:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763697.547664:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763697.547674:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763697.547679:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763697.547703:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496261504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763697.547742:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763697/real 1789763697] req@ffff8dd211834700 x1876699496261120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763697.547770:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496261120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763697.547786:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763697.547791:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763697.547796:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496261120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763697.547836:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496261248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763697.547885:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763697.547912:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763697/real 1789763697] req@ffff8dd211835c00 x1876699496261248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763697.547941:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496261248/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763697.547953:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763697.547957:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763697.547962:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496261248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763697.547970:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496261376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763697.547999:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763697.548013:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763697/real 1789763697] req@ffff8dd211836a00 x1876699496261376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763697.548028:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496261376/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763697.548039:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763697.548043:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763697.548047:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496261376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763697.548054:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496261504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763697.548086:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763697.548108:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763697/real 1789763697] req@ffff8dd211834000 x1876699496261504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763697.548133:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496261504/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763697.548165:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763697.548203:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763697.548209:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496261504:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763698.123413:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763699.147395:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763700.171434:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763701.195407:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763702.219394:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763702.667348:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763702.667361:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763702.667367:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763702.667380:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3682 00000100:00080000:2.0:1789763702.667393:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763702.667402:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763702.667447:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496261632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763702.667482:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763702.667487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763702.667490:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763702.667496:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3682 00000100:00080000:2.0:1789763702.667500:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763702.667506:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763702.667518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763702.667538:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496261632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763702.667539:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496261760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763702.667596:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763702.667601:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763702.667604:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763702.667610:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3682 00000100:00080000:2.0:1789763702.667615:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763702.667621:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763702.667630:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763702.667653:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496261888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763702.667665:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763702.667689:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763702.667697:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763702.667701:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763702.667712:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3682 00000100:00080000:2.0:1789763702.667721:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763702.667727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763702/real 1789763702] req@ffff8dd205264000 x1876699496261632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763702.667737:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763702.667758:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763702.667769:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496261632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763702.667785:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496262016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763702.667800:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763702.667808:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763702.667819:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496261632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763702.667862:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496261760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763702.667893:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763702.667908:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763702/real 1789763702] req@ffff8dd205265880 x1876699496261760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763702.667923:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496261760/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763702.667934:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763702.667938:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763702.667942:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496261760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763702.667950:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496261888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763702.667971:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763702.667982:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763702/real 1789763702] req@ffff8dd205267b80 x1876699496261888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763702.667997:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496261888/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763702.668007:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763702.668010:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763702.668014:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496261888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763702.668020:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496262016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763702.668039:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763702.668049:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763702/real 1789763702] req@ffff8dd205266a00 x1876699496262016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763702.668063:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496262016/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763702.668073:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763702.668076:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763702.668080:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496262016:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763703.243440:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763704.267422:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763705.291571:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763706.315291:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763707.339322:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763707.787360:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763707.787373:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763707.787379:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763707.787394:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3687 00000100:00080000:2.0:1789763707.787408:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763707.787419:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763707.787488:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496262144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763707.787516:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763707.787544:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763707.787547:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763707.787573:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3687 00000100:00080000:2.0:1789763707.787579:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763707.787585:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763707.787597:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496262144:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1789763707.787598:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763707.787627:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496262272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763707.787642:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763707.787647:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763707.787650:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763707.787656:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3687 00000100:00080000:2.0:1789763707.787661:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763707.787665:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763707.787672:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763707.787687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763707.787690:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496262400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763707.787706:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763707.787711:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763707.787713:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763707.787718:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3687 00000100:00080000:2.0:1789763707.787722:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763707.787727:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763707.787734:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763707.787747:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763707/real 1789763707] req@ffff8dd205264700 x1876699496262144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763707.787752:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496262528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763707.787813:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496262144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763707.787829:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763707.787835:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763707.787840:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496262144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763707.787870:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496262272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763707.787896:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763707.787911:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763707/real 1789763707] req@ffff8dd205265f80 x1876699496262272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763707.787926:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496262272/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763707.787936:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763707.787940:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763707.787944:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496262272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763707.787950:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496262400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763707.787968:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763707.787978:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763707/real 1789763707] req@ffff8dd205265180 x1876699496262400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763707.787991:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496262400/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763707.788009:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763707.788012:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763707.788015:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496262400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763707.788021:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496262528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763707.788039:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763707.788049:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763707/real 1789763707] req@ffff8dd205266d80 x1876699496262528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763707.788062:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496262528/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763707.788071:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763707.788074:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763707.788077:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496262528:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763708.363413:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763709.387439:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763710.411414:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763711.435374:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763712.459392:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763712.907258:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763712.907267:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763712.907270:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763712.907279:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3692 00000100:00080000:0.0:1789763712.907288:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763712.907293:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763712.907324:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496262656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763712.907346:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763712.907350:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763712.907353:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763712.907357:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3692 00000100:00080000:0.0:1789763712.907361:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763712.907365:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763712.907374:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763712.907384:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496262656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763712.907387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496262784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763712.907397:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763712.907400:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763712.907402:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763712.907405:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3692 00000100:00080000:0.0:1789763712.907408:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763712.907412:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763712.907416:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763712.907424:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496262912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763712.907433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763712.907436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763712.907438:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763712.907441:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3692 00000800:00000100:1.0:1789763712.907443:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763712.907444:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763712.907447:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763712.907451:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763712.907460:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496263040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763712.907496:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763712/real 1789763712] req@ffff8dd211835f80 x1876699496262656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763712.907517:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496262656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763712.907527:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763712.907531:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763712.907533:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496262656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763712.907555:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496262784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763712.907603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763712.907621:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763712/real 1789763712] req@ffff8dd211834380 x1876699496262784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763712.907637:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496262784/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763712.907647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763712.907652:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763712.907656:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496262784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763712.907663:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496262912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763712.907684:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763712.907714:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763712/real 1789763712] req@ffff8dd211836680 x1876699496262912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763712.907722:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496262912/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763712.907728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763712.907729:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763712.907731:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496262912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763712.907735:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496263040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763712.907746:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763712.907752:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763712/real 1789763712] req@ffff8dd211834a80 x1876699496263040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763712.907760:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496263040/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763712.907786:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763712.907789:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763712.907793:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496263040:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763713.483393:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763714.507384:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763715.531420:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763716.555388:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763717.579482:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763718.027310:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763718.027323:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763718.027330:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763718.027344:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3697 00000100:00080000:0.0:1789763718.027361:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763718.027373:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763718.027432:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496263168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763718.027499:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763718.027505:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763718.027507:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763718.027514:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3697 00000100:00080000:0.0:1789763718.027520:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763718.027528:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763718.027542:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763718.027561:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496263168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763718.027567:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496263296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763718.027587:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763718.027591:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763718.027595:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763718.027601:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3697 00000100:00080000:0.0:1789763718.027606:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763718.027611:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763718.027618:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763718.027633:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496263424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763718.027647:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763718.027652:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763718.027661:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763718.027662:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763718.027673:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3697 00000100:00080000:0.0:1789763718.027681:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763718.027687:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763718.027695:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763718.027712:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496263552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763718.027732:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763718/real 1789763718] req@ffff8dd211836d80 x1876699496263168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763718.027761:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496263168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763718.027776:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763718.027781:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763718.027786:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496263168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763718.027827:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496263296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763718.027893:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763718.027919:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763718/real 1789763718] req@ffff8dd211837480 x1876699496263296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763718.027944:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496263296/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763718.027975:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763718.027983:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763718.027991:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496263296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763718.028005:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496263424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763718.028045:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763718.028062:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763718/real 1789763718] req@ffff8dd211835180 x1876699496263424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763718.028078:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496263424/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763718.028089:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763718.028094:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763718.028098:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496263424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763718.028106:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496263552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763718.028127:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763718.028139:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763718/real 1789763718] req@ffff8dd211834e00 x1876699496263552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763718.028154:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496263552/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763718.028163:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763718.028198:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763718.028204:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496263552:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763718.603386:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763719.627495:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763720.651394:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763721.675409:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763722.699478:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763723.147437:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763723.147454:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763723.147461:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763723.147481:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3702 00000100:00080000:2.0:1789763723.147500:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763723.147512:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763723.147577:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496263680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763723.147626:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763723.147631:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763723.147634:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763723.147639:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3702 00000100:00080000:2.0:1789763723.147643:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763723.147648:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763723.147661:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763723.147683:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496263808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763723.147697:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763723.147702:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763723.147705:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763723.147709:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3702 00000100:00080000:2.0:1789763723.147713:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763723.147718:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763723.147726:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763723.147737:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496263680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763723.147745:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496263936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763723.147785:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763723.147790:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763723.147792:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763723.147799:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3702 00000100:00080000:2.0:1789763723.147803:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763723.147808:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763723.147813:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763723.147826:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496264064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763723.147891:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763723.147964:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763723/real 1789763723] req@ffff8dd205266300 x1876699496263680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763723.147999:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496263680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763723.148021:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763723.148030:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763723.148040:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496263680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763723.148077:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496263808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763723.148128:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763723.148157:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763723/real 1789763723] req@ffff8dd205266d80 x1876699496263808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763723.148255:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496263808/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763723.148274:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763723.148281:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763723.148290:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496263808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763723.148342:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496263936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763723.148392:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763723.148453:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763723/real 1789763723] req@ffff8dd205265180 x1876699496263936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763723.148482:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496263936/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763723.148500:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763723.148508:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763723.148515:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496263936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763723.148529:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496264064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763723.148574:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763723.148598:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763723/real 1789763723] req@ffff8dd205265f80 x1876699496264064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763723.148622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496264064/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763723.148639:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763723.148646:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763723.148653:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496264064:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763723.723463:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763724.747487:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763725.771483:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763726.795482:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763727.819486:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763728.267297:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763728.267311:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763728.267317:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763728.267330:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3707 00000100:00080000:2.0:1789763728.267345:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763728.267355:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763728.267435:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496264192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763728.267476:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763728.267481:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763728.267484:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763728.267490:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3707 00000100:00080000:2.0:1789763728.267495:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763728.267500:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763728.267514:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763728.267533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496264320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763728.267553:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496264192:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763728.267554:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763728.267572:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763728.267575:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763728.267582:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3707 00000100:00080000:2.0:1789763728.267585:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763728.267589:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763728.267596:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763728.267614:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496264448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763728.267628:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763728.267633:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763728.267636:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763728.267640:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3707 00000100:00080000:2.0:1789763728.267644:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763728.267649:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763728.267655:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763728.267664:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763728.267672:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496264576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763728.267751:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763728/real 1789763728] req@ffff8dd205264700 x1876699496264192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763728.267789:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496264192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763728.267805:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763728.267810:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763728.267816:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496264192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763728.267845:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496264320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763728.267880:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763728.267904:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763728/real 1789763728] req@ffff8dd205266a00 x1876699496264320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763728.267929:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496264320/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763728.267947:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763728.267954:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763728.267961:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496264320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763728.267975:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496264448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763728.268010:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763728.268043:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763728/real 1789763728] req@ffff8dd205267b80 x1876699496264448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763728.268083:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496264448/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763728.268101:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763728.268108:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763728.268115:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496264448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763728.268127:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496264576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763728.268205:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763728.268223:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763728/real 1789763728] req@ffff8dd205265880 x1876699496264576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763728.268239:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496264576/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763728.268249:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763728.268253:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763728.268257:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496264576:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763728.843423:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763729.867409:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763730.891400:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763731.915389:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763732.939408:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763733.387342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763733.387353:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763733.387358:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763733.387368:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3712 00000100:00080000:1.0:1789763733.387392:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763733.387399:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763733.387439:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496264704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763733.387470:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763733.387476:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763733.387478:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763733.387484:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3712 00000100:00080000:1.0:1789763733.387490:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763733.387496:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763733.387508:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763733.387527:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496264832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763733.387538:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496264704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763733.387543:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763733.387547:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763733.387550:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763733.387555:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3712 00000100:00080000:1.0:1789763733.387560:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763733.387565:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763733.387572:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763733.387585:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496264960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763733.387598:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763733.387603:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763733.387605:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763733.387610:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763733.387610:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3712 00000100:00080000:1.0:1789763733.387615:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763733.387619:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763733.387626:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763733.387639:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496265088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763733.387658:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763733/real 1789763733] req@ffff8dd20504d180 x1876699496264704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763733.387683:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496264704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763733.387699:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763733.387704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763733.387711:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496264704:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763733.387740:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496264832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763733.387766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763733.387780:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763733/real 1789763733] req@ffff8dd20504ce00 x1876699496264832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763733.387792:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496264832/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763733.387800:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763733.387803:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763733.387806:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496264832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763733.387811:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496264960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763733.387828:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763733.387836:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763733/real 1789763733] req@ffff8dd20504fb80 x1876699496264960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763733.387847:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496264960/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763733.387861:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763733.387864:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763733.387867:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496264960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763733.387872:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496265088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763733.387886:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763733.387894:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763733/real 1789763733] req@ffff8dd20504f800 x1876699496265088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763733.387904:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496265088/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763733.387912:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763733.387914:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763733.387917:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496265088:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763733.963470:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763734.987373:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763736.011447:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763737.035449:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763738.059321:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763738.507312:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763738.507339:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763738.507342:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763738.507352:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3718 00000100:00080000:1.0:1789763738.507361:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763738.507368:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763738.507419:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496265216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763738.507440:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763738.507443:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763738.507444:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763738.507448:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3718 00000100:00080000:1.0:1789763738.507450:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763738.507452:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763738.507459:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763738.507476:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496265344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763738.507487:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763738.507490:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496265216:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763738.507491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763738.507495:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763738.507502:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3718 00000100:00080000:1.0:1789763738.507505:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763738.507508:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763738.507511:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763738.507521:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496265472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763738.507530:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763738.507532:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763738.507534:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763738.507536:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3718 00000100:00080000:1.0:1789763738.507550:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763738.507553:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763738.507556:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763738.507562:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763738.507567:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496265600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763738.507609:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763738/real 1789763738] req@ffff8dd20504c700 x1876699496265216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763738.507658:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496265216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763738.507679:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763738.507686:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763738.507694:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496265216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763738.507747:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496265344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763738.507769:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763738.507781:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763738/real 1789763738] req@ffff8dd20504f100 x1876699496265344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763738.507793:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496265344/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763738.507835:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763738.507841:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763738.507848:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496265344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763738.507860:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496265472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763738.507895:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763738.507915:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763738/real 1789763738] req@ffff8dd20504ed80 x1876699496265472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763738.507936:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496265472/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763738.507966:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763738.507969:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763738.507972:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496265472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763738.507978:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496265600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763738.507994:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763738.508003:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763738/real 1789763738] req@ffff8dd20504d880 x1876699496265600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763738.508013:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496265600/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763738.508020:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763738.508023:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763738.508026:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496265600:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763739.083406:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763740.107461:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763741.131455:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763742.155442:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763743.179440:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763743.627329:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763743.627343:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763743.627348:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763743.627362:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3723 00000100:00080000:1.0:1789763743.627376:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763743.627385:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763743.627434:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496265728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763743.627483:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763743.627488:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763743.627491:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763743.627496:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3723 00000100:00080000:1.0:1789763743.627500:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763743.627504:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763743.627517:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763743.627533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496265856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763743.627547:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763743.627552:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763743.627554:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763743.627559:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3723 00000100:00080000:1.0:1789763743.627562:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763743.627567:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763743.627573:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763743.627580:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496265728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763743.627592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496265984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763743.627608:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763743.627613:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763743.627615:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763743.627619:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3723 00000100:00080000:1.0:1789763743.627623:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763743.627629:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763743.627635:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763743.627649:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496266112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763743.627729:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496265856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763743.627803:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496265984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763743.627855:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496266112:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763744.203476:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763745.227338:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763746.251453:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763746.699597:0:2344:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789763746.699668:0:2344:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763746.699695:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763746.699757:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763746.699774:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763746.699789:0:2344:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1789763746.699855:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763743/real 1789763746] req@ffff8dd20504ea00 x1876699496265984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.699904:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496265984/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.699929:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.699938:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763746.699946:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496265984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763746.699959:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763743/real 1789763746] req@ffff8dd20504c380 x1876699496266112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.699986:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496266112/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.700003:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.700010:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763746.700018:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496266112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763746.700032:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763743/real 1789763746] req@ffff8dd20504c000 x1876699496265728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.700059:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496265728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.700076:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.700082:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763746.700090:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496265728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763746.700103:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763743/real 1789763746] req@ffff8dd20504ca80 x1876699496265856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763746.700128:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496265856/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763746.700144:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763746.700151:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763746.700158:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496265856:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763747.275425:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763748.299406:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763748.747375:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763748.747389:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763748.747394:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763748.747407:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3728 00000100:00080000:1.0:1789763748.747421:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763748.747429:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763748.747477:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504dc00 x1876699496266240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763748.747515:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763748.747521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763748.747524:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763748.747529:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3728 00000100:00080000:1.0:1789763748.747534:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763748.747538:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763748.747550:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763748.747574:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496266368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763748.747598:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763748.747613:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496266240:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763748.747614:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763748.747622:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763748.747642:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3728 00000100:00080000:1.0:1789763748.747652:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763748.747675:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763748.747686:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763748.747728:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496266496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763748.747754:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763748.747761:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763748.747766:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763748.747775:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3728 00000800:00000100:0.0:1789763748.747791:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763748.747794:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763748.747811:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763748.747825:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763748.747857:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496266624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763748.747916:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763748/real 1789763748] req@ffff8dd20504dc00 x1876699496266240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763748.747963:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504dc00 x1876699496266240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763748.747988:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763748.747996:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763748.748002:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496266240:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763748.748051:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496266368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763748.748109:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763748.748137:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763748/real 1789763748] req@ffff8dd20504ca80 x1876699496266368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763748.748164:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496266368/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763748.748279:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763748.748290:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763748.748299:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496266368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763748.748319:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496266496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763748.748370:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763748.748397:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763748/real 1789763748] req@ffff8dd20504c000 x1876699496266496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763748.748423:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496266496/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763748.748441:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763748.748449:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763748.748457:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496266496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763748.748472:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496266624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763748.748517:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763748.748542:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763748/real 1789763748] req@ffff8dd20504c380 x1876699496266624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763748.748566:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496266624/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763748.748582:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763748.748589:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763748.748597:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496266624:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763749.323413:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763750.347386:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763751.371388:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763752.395408:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763753.419436:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763753.867342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763753.867358:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763753.867363:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763753.867374:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3733 00000100:00080000:2.0:1789763753.867387:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763753.867395:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763753.867435:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496266752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763753.867468:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763753.867472:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763753.867474:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763753.867479:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3733 00000100:00080000:2.0:1789763753.867482:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763753.867486:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763753.867496:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763753.867512:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496266880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763753.867524:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763753.867539:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496266752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763753.867550:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763753.867553:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763753.867559:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3733 00000100:00080000:2.0:1789763753.867563:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763753.867567:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763753.867573:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763753.867587:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496267008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763753.867605:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763753.867609:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763753.867611:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763753.867618:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763753.867623:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3733 00000100:00080000:2.0:1789763753.867630:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763753.867635:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763753.867641:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763753.867654:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496267136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763753.867674:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763753/real 1789763753] req@ffff8dd205264000 x1876699496266752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763753.867707:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496266752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763753.867721:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763753.867725:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763753.867730:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496266752:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763753.867755:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496266880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763753.867789:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763753.867804:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763753/real 1789763753] req@ffff8dd205267800 x1876699496266880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763753.867817:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496266880/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763753.867826:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763753.867830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763753.867834:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496266880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763753.867840:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496267008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763753.867859:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763753.867869:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763753/real 1789763753] req@ffff8dd205264e00 x1876699496267008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763753.867881:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496267008/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763753.867898:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763753.867901:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763753.867904:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496267008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763753.867909:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496267136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763753.867925:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763753.867934:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763753/real 1789763753] req@ffff8dd205264380 x1876699496267136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763753.867946:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496267136/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763753.867954:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763753.867956:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763753.867959:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496267136:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763754.443432:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763755.467434:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763756.491424:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763757.515401:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763758.539375:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763758.987354:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763758.987370:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763758.987379:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763758.987397:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3738 00000100:00080000:2.0:1789763758.987413:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763758.987425:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763758.987479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496267264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763758.987525:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763758.987533:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763758.987537:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763758.987545:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3738 00000100:00080000:2.0:1789763758.987551:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763758.987559:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763758.987578:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763758.987622:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496267392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763758.987629:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496267264:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763758.987658:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763758.987665:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763758.987668:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763758.987677:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3738 00000100:00080000:2.0:1789763758.987683:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763758.987706:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763758.987717:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763758.987740:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496267520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763758.987754:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763758.987773:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763758.987781:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763758.987786:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763758.987796:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3738 00000100:00080000:2.0:1789763758.987804:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763758.987827:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763758/real 1789763758] req@ffff8dd205267100 x1876699496267264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763758.987833:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763758.987851:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763758.987887:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496267648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763758.987890:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496267264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763758.987953:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763758.987962:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763758.987970:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496267264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763758.988021:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496267392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763758.988053:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763758.988093:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763758/real 1789763758] req@ffff8dd205266680 x1876699496267392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763758.988120:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496267392/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763758.988138:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763758.988214:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763758.988224:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496267392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763758.988238:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496267520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1789763758.988290:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763758.988310:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763758/real 1789763758] req@ffff8dd205265500 x1876699496267520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763758.988326:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496267520/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763758.988338:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763758.988342:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763758.988346:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496267520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1789763758.988354:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496267648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1789763758.988378:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763758.988389:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763758/real 1789763758] req@ffff8dd205264a80 x1876699496267648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763758.988404:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496267648/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763758.988414:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763758.988417:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1789763758.988421:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496267648:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763759.563425:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763760.587447:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763761.611331:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763762.635408:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763763.659345:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763764.107301:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763764.107321:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763764.107327:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763764.107340:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3743 00000100:00080000:0.0:1789763764.107355:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763764.107363:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763764.107412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496267776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763764.107450:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763764.107455:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763764.107458:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763764.107463:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3743 00000100:00080000:0.0:1789763764.107467:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763764.107471:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763764.107483:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763764.107499:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496267904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763764.107514:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763764.107520:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496267776:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763764.107521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763764.107528:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763764.107543:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3743 00000100:00080000:0.0:1789763764.107547:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763764.107551:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763764.107559:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763764.107575:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496268032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763764.107591:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763764.107595:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763764.107598:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763764.107602:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3743 00000100:00080000:0.0:1789763764.107618:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763764.107625:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763764.107633:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763764.107635:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763764.107654:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496268160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763764.107718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763764/real 1789763764] req@ffff8dd211834e00 x1876699496267776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763764.107746:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496267776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763764.107761:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763764.107766:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763764.107771:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496267776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763764.107803:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496267904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763764.107838:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763764.107855:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763764/real 1789763764] req@ffff8dd211835180 x1876699496267904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763764.107889:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496267904/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763764.107900:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763764.107906:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763764.107914:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496267904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763764.107927:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496268032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763764.107967:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763764.107991:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763764/real 1789763764] req@ffff8dd211837480 x1876699496268032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763764.108016:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496268032/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763764.108029:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763764.108035:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763764.108042:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496268032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763764.108055:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496268160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763764.108097:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763764.108148:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763764/real 1789763764] req@ffff8dd211836d80 x1876699496268160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763764.108225:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496268160/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763764.108243:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763764.108250:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763764.108255:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496268160:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763764.683396:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763765.707389:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763766.731296:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763767.755296:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763768.779402:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763769.227297:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763769.227311:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763769.227317:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763769.227330:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3748 00000100:00080000:0.0:1789763769.227344:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763769.227354:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763769.227406:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496268288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763769.227469:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763769.227477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763769.227483:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763769.227493:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3748 00000100:00080000:0.0:1789763769.227502:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763769.227510:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763769.227525:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763769.227529:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496268288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763769.227550:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496268416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763769.227567:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763769.227572:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763769.227574:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763769.227580:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3748 00000100:00080000:0.0:1789763769.227586:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763769.227590:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763769.227597:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763769.227621:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496268544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763769.227634:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763769.227639:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763769.227642:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763769.227646:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3748 00000100:00080000:0.0:1789763769.227650:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763769.227654:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763769.227661:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763769.227668:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763769.227682:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496268672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763769.227763:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763769/real 1789763769] req@ffff8dd211834a80 x1876699496268288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763769.227789:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496268288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763769.227804:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763769.227809:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763769.227814:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496268288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763769.227843:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496268416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763769.227893:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763769.227920:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763769/real 1789763769] req@ffff8dd211836680 x1876699496268416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763769.227959:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496268416/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763769.227978:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763769.227985:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763769.227992:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496268416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763769.228006:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496268544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763769.228044:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763769.228062:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763769/real 1789763769] req@ffff8dd211834380 x1876699496268544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763769.228108:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496268544/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763769.228120:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763769.228124:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763769.228128:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496268544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763769.228137:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496268672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763769.228165:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763769.228214:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763769/real 1789763769] req@ffff8dd211835f80 x1876699496268672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763769.228230:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496268672/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763769.228241:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763769.228245:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763769.228250:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496268672:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763769.803320:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763770.827283:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763771.851326:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763772.875287:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763773.899289:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763774.347268:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763774.347278:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763774.347282:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763774.347290:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3753 00000100:00080000:2.0:1789763774.347299:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763774.347305:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763774.347338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496268800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763774.347360:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763774.347362:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763774.347364:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763774.347367:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3753 00000100:00080000:2.0:1789763774.347369:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763774.347371:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763774.347379:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763774.347389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496268928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763774.347398:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496268800:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763774.347399:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763774.347408:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763774.347410:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763774.347413:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3753 00000100:00080000:2.0:1789763774.347415:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763774.347417:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763774.347421:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763774.347429:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496269056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763774.347441:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763774.347444:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763774.347445:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763774.347447:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3753 00000100:00080000:2.0:1789763774.347449:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763774.347452:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763774.347455:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763774.347463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496269184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763774.347470:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763774.347510:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763774/real 1789763774] req@ffff8dd205267480 x1876699496268800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763774.347524:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496268800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763774.347531:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763774.347534:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763774.347537:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496268800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763774.347554:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496268928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763774.347572:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763774.347580:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763774/real 1789763774] req@ffff8dd205264a80 x1876699496268928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763774.347588:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496268928/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763774.347593:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763774.347595:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763774.347598:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496268928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763774.347602:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496269056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763774.347614:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763774.347619:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763774/real 1789763774] req@ffff8dd205265500 x1876699496269056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763774.347626:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496269056/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763774.347633:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763774.347636:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763774.347639:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496269056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763774.347645:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496269184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763774.347665:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763774.347677:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763774/real 1789763774] req@ffff8dd205266680 x1876699496269184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763774.347690:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496269184/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763774.347695:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763774.347698:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763774.347700:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496269184:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763774.923381:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763775.947302:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763776.971435:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763777.995375:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763779.019341:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763779.467261:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763779.467271:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763779.467275:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763779.467285:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3759 00000100:00080000:2.0:1789763779.467296:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763779.467304:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763779.467342:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496269312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763779.467369:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763779.467373:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763779.467375:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763779.467379:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3759 00000100:00080000:2.0:1789763779.467383:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763779.467387:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763779.467396:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763779.467404:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496269312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763779.467412:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496269440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763779.467426:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763779.467429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763779.467431:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763779.467436:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3759 00000100:00080000:2.0:1789763779.467441:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763779.467446:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763779.467452:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763779.467466:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496269568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763779.467467:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763779.467477:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763779.467481:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763779.467482:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763779.467487:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3759 00000100:00080000:2.0:1789763779.467490:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763779.467492:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763779.467496:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763779.467511:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763779/real 1789763779] req@ffff8dd205267100 x1876699496269312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763779.467511:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496269696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763779.467554:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496269312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763779.467563:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763779.467566:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763779.467569:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496269312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763779.467587:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496269440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763779.467603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763779.467611:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763779/real 1789763779] req@ffff8dd205264380 x1876699496269440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763779.467619:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496269440/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763779.467625:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763779.467627:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763779.467639:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496269440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763779.467643:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496269568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763779.467656:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763779.467662:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763779/real 1789763779] req@ffff8dd205264e00 x1876699496269568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763779.467674:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496269568/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763779.467680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763779.467681:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763779.467683:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496269568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763779.467687:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496269696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763779.467697:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763779.467703:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763779/real 1789763779] req@ffff8dd205267800 x1876699496269696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763779.467711:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496269696/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763779.467715:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763779.467717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763779.467719:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496269696:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789763780.043275:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763781.067348:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763782.091307:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763783.115469:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763784.139297:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763784.587251:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763784.587259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763784.587263:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763784.587270:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3764 00000100:00080000:0.0:1789763784.587280:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763784.587288:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763784.587334:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496269824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763784.587355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763784.587357:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763784.587359:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763784.587362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3764 00000100:00080000:0.0:1789763784.587365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763784.587367:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763784.587376:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763784.587387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496269952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763784.587395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763784.587398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763784.587402:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763784.587402:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496269824:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763784.587412:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3764 00000100:00080000:0.0:1789763784.587416:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763784.587421:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763784.587427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763784.587443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496270080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763784.587453:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763784.587456:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763784.587457:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763784.587461:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3764 00000100:00080000:0.0:1789763784.587464:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763784.587467:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763784.587471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763784.587472:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763784.587484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496270208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763784.587519:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763784/real 1789763784] req@ffff8dd211834000 x1876699496269824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763784.587537:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496269824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763784.587549:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763784.587553:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763784.587557:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496269824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763784.587580:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496269952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763784.587603:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763784.587616:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763784/real 1789763784] req@ffff8dd211836a00 x1876699496269952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763784.587629:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496269952/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763784.587639:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763784.587642:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763784.587646:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496269952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763784.587653:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496270080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763784.587673:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763784.587684:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763784/real 1789763784] req@ffff8dd211835c00 x1876699496270080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763784.587697:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496270080/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763784.587706:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763784.587710:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763784.587724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496270080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763784.587730:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496270208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763784.587750:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763784.587762:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763784/real 1789763784] req@ffff8dd211834700 x1876699496270208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763784.587774:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496270208/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763784.587783:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763784.587786:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763784.587789:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496270208:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763785.163289:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763786.187295:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763787.211345:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763788.235331:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763789.259344:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763789.707305:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763789.707316:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763789.707321:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763789.707332:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3769 00000100:00080000:0.0:1789763789.707343:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763789.707351:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763789.707390:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496270336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763789.707417:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763789.707421:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763789.707423:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763789.707427:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3769 00000100:00080000:0.0:1789763789.707430:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763789.707433:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763789.707443:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763789.707454:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496270464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763789.707465:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763789.707470:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763789.707470:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496270336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763789.707474:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763789.707482:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3769 00000100:00080000:0.0:1789763789.707487:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763789.707491:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763789.707496:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763789.707508:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496270592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763789.707517:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763789.707521:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763789.707522:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763789.707527:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3769 00000100:00080000:0.0:1789763789.707539:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763789.707543:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763789.707548:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763789.707552:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763789.707562:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496270720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763789.707606:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763789/real 1789763789] req@ffff8dd211837800 x1876699496270336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763789.707634:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496270336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763789.707649:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763789.707654:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763789.707658:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496270336:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763789.707716:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496270464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763789.707747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763789.707763:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763789/real 1789763789] req@ffff8dd211837b80 x1876699496270464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763789.707778:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496270464/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763789.707787:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763789.707791:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763789.707795:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496270464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763789.707802:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496270592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763789.707820:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763789.707831:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763789/real 1789763789] req@ffff8dd211837100 x1876699496270592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763789.707843:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496270592/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763789.707852:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763789.707856:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763789.707859:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496270592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763789.707864:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496270720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763789.707890:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763789.707937:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763789/real 1789763789] req@ffff8dd211836300 x1876699496270720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763789.707955:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496270720/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763789.707968:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763789.707973:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763789.707978:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496270720:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763790.283305:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763791.307315:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763792.331312:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763793.355326:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763794.379557:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763794.827251:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763794.827259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763794.827262:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763794.827270:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3774 00000100:00080000:2.0:1789763794.827279:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763794.827286:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763794.827319:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496270848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763794.827352:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763794.827355:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763794.827356:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763794.827359:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3774 00000100:00080000:2.0:1789763794.827361:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763794.827364:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763794.827372:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763794.827381:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496270976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763794.827389:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763794.827393:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763794.827395:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496270848:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763794.827395:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763794.827402:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3774 00000100:00080000:2.0:1789763794.827407:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763794.827410:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763794.827414:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763794.827433:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496271104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763794.827446:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763794.827448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763794.827450:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763794.827452:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3774 00000100:00080000:2.0:1789763794.827455:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763794.827457:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763794.827460:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763794.827468:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496271232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763794.827481:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763794.827525:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763794/real 1789763794] req@ffff8dd205264000 x1876699496270848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763794.827544:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496270848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763794.827555:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763794.827559:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763794.827563:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496270848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763794.827586:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496270976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763794.827623:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763794.827636:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763794/real 1789763794] req@ffff8dd205265880 x1876699496270976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763794.827648:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496270976/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763794.827656:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763794.827660:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763794.827675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496270976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763794.827681:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496271104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763794.827702:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763794.827714:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763794/real 1789763794] req@ffff8dd205267b80 x1876699496271104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763794.827726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496271104/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763794.827744:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763794.827747:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763794.827750:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496271104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763794.827756:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496271232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763794.827776:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763794.827788:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763794/real 1789763794] req@ffff8dd205266a00 x1876699496271232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763794.827811:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496271232/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763794.827820:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763794.827824:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763794.827827:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496271232:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763795.403307:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763796.427281:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763797.451283:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763798.475295:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763799.499321:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763799.947283:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763799.947292:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763799.947295:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763799.947304:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3779 00000100:00080000:0.0:1789763799.947313:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763799.947318:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763799.947353:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496271360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763799.947373:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763799.947376:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763799.947377:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763799.947380:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3779 00000100:00080000:0.0:1789763799.947382:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763799.947384:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763799.947392:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763799.947401:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496271488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763799.947408:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763799.947411:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763799.947420:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763799.947422:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496271360:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763799.947424:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3779 00000100:00080000:0.0:1789763799.947430:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763799.947435:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763799.947439:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763799.947449:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496271616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763799.947456:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763799.947459:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763799.947461:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763799.947464:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3779 00000100:00080000:0.0:1789763799.947481:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763799.947484:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763799.947487:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763799.947494:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763799.947496:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496271744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763799.947535:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763799/real 1789763799] req@ffff8dd211836300 x1876699496271360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763799.947549:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496271360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763799.947557:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763799.947560:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763799.947563:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496271360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763799.947569:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496271488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763799.947587:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763799.947595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763799/real 1789763799] req@ffff8dd211837100 x1876699496271488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763799.947608:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496271488/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763799.947617:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763799.947620:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763799.947624:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496271488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763799.947650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496271616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763799.947667:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763799.947675:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763799/real 1789763799] req@ffff8dd211837b80 x1876699496271616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763799.947688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496271616/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763799.947693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763799.947695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763799.947697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496271616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763799.947702:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496271744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763799.947712:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763799.947718:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763799/real 1789763799] req@ffff8dd211837800 x1876699496271744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763799.947726:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496271744/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763799.947731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763799.947733:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763799.947734:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496271744:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789763800.523329:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763801.547317:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763802.571312:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763803.595270:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763804.619318:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763805.067279:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763805.067291:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763805.067295:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763805.067306:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3784 00000100:00080000:3.0:1789763805.067318:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763805.067326:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763805.067361:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144380 x1876699496271872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763805.067387:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763805.067391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763805.067393:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763805.067397:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3784 00000100:00080000:3.0:1789763805.067401:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763805.067406:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763805.067415:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763805.067419:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496271872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763805.067432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145500 x1876699496272000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763805.067444:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763805.067448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763805.067450:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763805.067454:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3784 00000100:00080000:3.0:1789763805.067458:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763805.067462:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763805.067467:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763805.067479:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144a80 x1876699496272128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763805.067489:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789763805.067495:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763805.067497:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763805.067501:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763805.067507:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3784 00000100:00080000:3.0:1789763805.067512:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763805.067517:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763805.067522:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763805.067537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145f80 x1876699496272256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763805.067543:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763805/real 1789763805] req@ffff8dd203144380 x1876699496271872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763805.067579:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144380 x1876699496271872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763805.067591:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763805.067596:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763805.067601:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496271872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763805.067623:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496272000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763805.067645:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763805.067657:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763805/real 1789763805] req@ffff8dd203145500 x1876699496272000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763805.067670:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145500 x1876699496272000/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763805.067679:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763805.067683:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763805.067687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496272000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763805.067693:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496272128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763805.067710:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763805.067720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763805/real 1789763805] req@ffff8dd203144a80 x1876699496272128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763805.067732:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144a80 x1876699496272128/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763805.067747:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763805.067750:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763805.067754:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496272128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763805.067759:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496272256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763805.067776:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763805.067785:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763805/real 1789763805] req@ffff8dd203145f80 x1876699496272256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763805.067797:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145f80 x1876699496272256/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763805.067820:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763805.067823:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763805.067827:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496272256:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763805.643304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763806.667288:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763807.691282:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763808.715296:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763809.739361:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763810.187294:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763810.187306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763810.187310:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763810.187322:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3789 00000100:00080000:0.0:1789763810.187335:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763810.187344:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763810.187391:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496272384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763810.187417:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763810.187420:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763810.187421:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763810.187425:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3789 00000100:00080000:0.0:1789763810.187427:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763810.187429:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763810.187441:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763810.187453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496272512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763810.187460:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496272384:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763810.187463:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763810.187469:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763810.187471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763810.187475:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3789 00000100:00080000:0.0:1789763810.187477:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763810.187480:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763810.187483:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763810.187493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496272640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763810.187500:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763810.187502:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763810.187503:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763810.187506:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3789 00000100:00080000:0.0:1789763810.187529:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763810.187531:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763810.187535:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763810.187543:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496272768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763810.187600:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496272512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763810.187635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496272640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763810.187662:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496272768:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763810.763448:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763811.787319:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763812.811342:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763813.259442:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1789763813.259487:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763813.259503:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763813.259540:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763813.259551:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1789763813.259564:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789763813.259591:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763810/real 1789763813] req@ffff8dd211835c00 x1876699496272512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763813.259621:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496272512/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763813.259634:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763813.259639:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763813.259644:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496272512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763813.259652:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763810/real 1789763813] req@ffff8dd211836a00 x1876699496272640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763813.259665:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496272640/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763813.259673:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763813.259677:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763813.259686:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496272640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763813.259694:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763810/real 1789763813] req@ffff8dd211834000 x1876699496272768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763813.259705:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496272768/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763813.259713:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763813.259717:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763813.259720:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496272768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763813.259731:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763810/real 1789763813] req@ffff8dd211834700 x1876699496272384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763813.259760:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496272384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763813.259769:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763813.259772:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763813.259776:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496272384:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789763813.835402:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763814.859332:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763815.307326:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763815.307335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763815.307339:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763815.307350:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3794 00000100:00080000:2.0:1789763815.307366:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763815.307375:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763815.307409:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496272896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763815.307431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763815.307434:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763815.307436:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763815.307439:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3794 00000100:00080000:2.0:1789763815.307442:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763815.307445:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763815.307467:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763815.307482:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496272896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763815.307483:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496273024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763815.307501:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763815.307504:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763815.307506:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763815.307511:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3794 00000100:00080000:2.0:1789763815.307515:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763815.307519:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763815.307525:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763815.307537:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496273152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763815.307547:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763815.307551:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763815.307553:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763815.307559:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3794 00000100:00080000:2.0:1789763815.307563:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763815.307568:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763815.307568:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763815.307577:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763815.307589:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496273280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763815.307647:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763815/real 1789763815] req@ffff8dd205264700 x1876699496272896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763815.307688:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496272896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763815.307700:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763815.307704:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763815.307708:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496272896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763815.307751:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496273024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763815.307779:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763815.307793:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763815/real 1789763815] req@ffff8dd205265f80 x1876699496273024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763815.307802:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496273024/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763815.307812:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763815.307815:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763815.307819:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496273024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763815.307826:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496273152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763815.307848:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763815.307860:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763815/real 1789763815] req@ffff8dd205265180 x1876699496273152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763815.307871:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496273152/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763815.307880:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763815.307882:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763815.307886:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496273152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763815.307892:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496273280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763815.307912:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763815.307923:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763815/real 1789763815] req@ffff8dd205266d80 x1876699496273280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763815.307936:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496273280/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763815.307944:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763815.307947:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763815.307951:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496273280:192.168.203.109@tcp:38:kworker.0 00010000:00010000:1.0:1789763815.883313:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763816.907416:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763817.931372:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763818.955422:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763819.979364:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763820.427283:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763820.427298:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763820.427303:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763820.427317:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3800 00000100:00080000:0.0:1789763820.427332:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763820.427342:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763820.427396:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496273408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763820.427424:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763820.427429:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763820.427432:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763820.427439:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3800 00000100:00080000:0.0:1789763820.427444:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763820.427450:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763820.427458:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496273408:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789763820.427463:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763820.427479:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496273536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763820.427496:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763820.427499:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763820.427501:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763820.427505:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3800 00000100:00080000:0.0:1789763820.427509:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763820.427513:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763820.427518:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763820.427527:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496273664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763820.427535:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763820.427538:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763820.427539:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763820.427541:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3800 00000100:00080000:0.0:1789763820.427543:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763820.427545:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763820.427548:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763820.427553:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763820.427558:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496273792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763820.427608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763820/real 1789763820] req@ffff8dd211835f80 x1876699496273408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763820.427625:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496273408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763820.427635:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763820.427638:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763820.427642:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496273408:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763820.427667:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496273536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763820.427687:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763820.427714:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763820/real 1789763820] req@ffff8dd211834380 x1876699496273536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763820.427723:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496273536/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763820.427740:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763820.427743:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763820.427745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496273536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763820.427749:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496273664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763820.427766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763820.427773:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763820/real 1789763820] req@ffff8dd211836680 x1876699496273664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763820.427781:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496273664/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763820.427786:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763820.427788:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763820.427790:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496273664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763820.427794:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496273792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763820.427818:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763820.427829:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763820/real 1789763820] req@ffff8dd211834a80 x1876699496273792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763820.427841:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496273792/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763820.427850:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763820.427854:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763820.427859:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496273792:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763821.003280:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763822.027400:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763823.051320:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763824.075379:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763825.099314:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763825.547286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763825.547297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763825.547302:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763825.547313:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3805 00000100:00080000:2.0:1789763825.547327:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763825.547333:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763825.547378:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496273920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763825.547406:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763825.547408:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763825.547410:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763825.547413:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3805 00000100:00080000:2.0:1789763825.547416:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763825.547418:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763825.547429:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763825.547443:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496274048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763825.547453:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496273920:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763825.547456:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763825.547466:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763825.547468:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763825.547478:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3805 00000100:00080000:2.0:1789763825.547480:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763825.547482:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763825.547486:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763825.547495:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496274176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763825.547507:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763825.547510:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763825.547511:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763825.547516:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3805 00000100:00080000:2.0:1789763825.547518:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763825.547521:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763825.547524:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763825.547533:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496274304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763825.547547:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763825.547595:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763825/real 1789763825] req@ffff8dd205266300 x1876699496273920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763825.547625:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496273920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763825.547637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763825.547642:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763825.547645:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496273920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763825.547682:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496274048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763825.547704:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763825.547713:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763825/real 1789763825] req@ffff8dd205266d80 x1876699496274048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763825.547722:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496274048/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763825.547728:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763825.547731:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763825.547733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496274048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763825.547737:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496274176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763825.547749:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763825.547755:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763825/real 1789763825] req@ffff8dd205265180 x1876699496274176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763825.547763:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496274176/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763825.547772:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763825.547774:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763825.547776:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496274176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763825.547780:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496274304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763825.547790:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763825.547796:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763825/real 1789763825] req@ffff8dd205265f80 x1876699496274304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763825.547803:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496274304/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763825.547808:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763825.547810:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763825.547812:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496274304:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763826.123409:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763827.147377:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763828.171282:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763829.195406:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763830.219320:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763830.667285:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763830.667295:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763830.667300:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763830.667311:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3810 00000100:00080000:2.0:1789763830.667323:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763830.667342:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763830.667405:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496274432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763830.667431:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763830.667434:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763830.667436:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763830.667439:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3810 00000100:00080000:2.0:1789763830.667441:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763830.667444:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763830.667466:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763830.667493:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496274560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763830.667503:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763830.667507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763830.667511:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763830.667526:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3810 00000100:00080000:2.0:1789763830.667528:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763830.667529:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496274432:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763830.667532:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763830.667537:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763830.667565:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496274688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763830.667573:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763830.667577:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763830.667578:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763830.667582:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3810 00000100:00080000:2.0:1789763830.667585:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763830.667588:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763830.667592:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763830.667601:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496274816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763830.667666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763830.667705:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763830/real 1789763830] req@ffff8dd205264700 x1876699496274432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763830.667719:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496274432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763830.667727:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763830.667730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763830.667733:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496274432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763830.667753:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496274560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763830.667769:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763830.667777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763830/real 1789763830] req@ffff8dd205266a00 x1876699496274560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763830.667785:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496274560/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763830.667790:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763830.667792:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763830.667794:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496274560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763830.667798:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496274688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763830.667809:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763830.667815:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763830/real 1789763830] req@ffff8dd205267b80 x1876699496274688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763830.667822:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496274688/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763830.667828:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763830.667830:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763830.667831:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496274688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763830.667835:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496274816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763830.667845:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763830.667851:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763830/real 1789763830] req@ffff8dd205265880 x1876699496274816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763830.667859:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496274816/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763830.667864:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763830.667865:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763830.667867:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496274816:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763831.243336:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763832.267376:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763833.291378:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763834.315377:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763835.339359:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763835.787370:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763835.787387:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763835.787394:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763835.787414:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3815 00000100:00080000:3.0:1789763835.787433:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763835.787448:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763835.787510:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147800 x1876699496274944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763835.787556:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763835.787597:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763835.787613:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763835.787621:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3815 00000100:00080000:3.0:1789763835.787628:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763835.787636:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763835.787654:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763835.787702:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496274944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763835.787703:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145c00 x1876699496275072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763835.787742:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763835.787748:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763835.787752:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763835.787760:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3815 00000100:00080000:3.0:1789763835.787766:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763835.787773:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763835.787782:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763835.787800:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203146300 x1876699496275200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763835.787813:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763835.787818:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763835.787819:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763835.787824:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3815 00000100:00080000:3.0:1789763835.787829:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763835.787833:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763835.787839:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763835.787842:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763835.787860:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144700 x1876699496275328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763835.787932:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763835/real 1789763835] req@ffff8dd203147800 x1876699496274944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763835.787957:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147800 x1876699496274944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763835.787971:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763835.787977:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763835.787982:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496274944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763835.788013:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496275072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763835.788043:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763835.788057:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763835/real 1789763835] req@ffff8dd203145c00 x1876699496275072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763835.788072:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145c00 x1876699496275072/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763835.788090:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763835.788094:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763835.788098:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496275072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763835.788105:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496275200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763835.788123:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763835.788134:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763835/real 1789763835] req@ffff8dd203146300 x1876699496275200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763835.788147:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203146300 x1876699496275200/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763835.788157:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763835.788160:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763835.788164:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203146300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496275200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763835.788171:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496275328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763835.788274:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763835.788291:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763835/real 1789763835] req@ffff8dd203144700 x1876699496275328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763835.788309:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144700 x1876699496275328/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763835.788326:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763835.788332:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763835.788339:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496275328:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763836.363380:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763837.387371:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763838.411338:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763839.435360:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763840.459338:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763840.907362:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763840.907375:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763840.907381:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763840.907394:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3820 00000100:00080000:0.0:1789763840.907408:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763840.907417:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763840.907466:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496275456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763840.907510:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763840.907515:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763840.907518:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763840.907523:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3820 00000100:00080000:0.0:1789763840.907527:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763840.907532:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763840.907570:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763840.907596:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496275584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763840.907603:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496275456:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763840.907634:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763840.907641:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763840.907644:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763840.907652:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3820 00000100:00080000:0.0:1789763840.907658:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763840.907664:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763840.907672:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763840.907689:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496275712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763840.907703:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763840.907712:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789763840.907713:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763840.907716:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763840.907726:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3820 00000100:00080000:0.0:1789763840.907733:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763840.907739:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763840.907745:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763840.907761:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496275840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763840.907777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763840/real 1789763840] req@ffff8dd211836d80 x1876699496275456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763840.907807:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496275456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763840.907822:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763840.907828:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763840.907833:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496275456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763840.907870:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496275584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763840.907917:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763840.907944:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763840/real 1789763840] req@ffff8dd211837480 x1876699496275584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763840.907968:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496275584/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763840.907979:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763840.907984:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763840.907988:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496275584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763840.907997:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496275712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763840.908026:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763840.908040:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763840/real 1789763840] req@ffff8dd211835180 x1876699496275712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763840.908064:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496275712/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763840.908074:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763840.908077:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763840.908081:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496275712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763840.908088:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496275840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763840.908108:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763840.908120:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763840/real 1789763840] req@ffff8dd211834e00 x1876699496275840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763840.908134:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496275840/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763840.908149:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763840.908156:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763840.908163:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496275840:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763841.483462:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763842.507414:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763843.531451:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763844.555324:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763845.579422:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763846.027342:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763846.027355:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763846.027361:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763846.027374:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3825 00000100:00080000:2.0:1789763846.027387:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763846.027396:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763846.027442:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496275968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763846.027486:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763846.027491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763846.027493:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763846.027498:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3825 00000100:00080000:2.0:1789763846.027525:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763846.027531:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763846.027544:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763846.027563:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496276096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763846.027577:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763846.027582:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763846.027585:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763846.027591:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3825 00000100:00080000:2.0:1789763846.027595:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763846.027600:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763846.027606:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763846.027619:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496276224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763846.027634:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763846.027640:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789763846.027641:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496275968:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763846.027644:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763846.027657:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3825 00000100:00080000:2.0:1789763846.027664:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763846.027669:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763846.027676:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763846.027692:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496276352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763846.027743:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763846.027793:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763846/real 1789763846] req@ffff8dd205264000 x1876699496275968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763846.027827:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496275968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763846.027848:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763846.027856:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763846.027864:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496275968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763846.027879:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496276096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763846.027919:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763846.027938:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763846/real 1789763846] req@ffff8dd205267800 x1876699496276096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763846.027958:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496276096/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763846.027975:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763846.027981:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763846.027988:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496276096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763846.028029:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496276224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763846.028069:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763846.028087:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763846/real 1789763846] req@ffff8dd205264e00 x1876699496276224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763846.028103:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496276224/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763846.028114:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763846.028118:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763846.028122:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496276224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763846.028129:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496276352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763846.028151:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763846.028162:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763846/real 1789763846] req@ffff8dd205264380 x1876699496276352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763846.028224:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496276352/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763846.028235:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763846.028239:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763846.028243:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496276352:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763846.603312:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763847.627330:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763848.651343:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763849.675389:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763850.699369:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763851.147350:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763851.147364:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763851.147370:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763851.147383:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3830 00000100:00080000:0.0:1789763851.147396:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763851.147405:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763851.147455:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496276480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763851.147521:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763851.147526:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763851.147528:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763851.147534:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3830 00000100:00080000:0.0:1789763851.147539:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763851.147545:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763851.147568:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763851.147592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496276608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763851.147597:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496276480:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763851.147625:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763851.147630:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763851.147632:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763851.147639:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3830 00000100:00080000:0.0:1789763851.147644:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763851.147648:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763851.147657:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763851.147675:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496276736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763851.147689:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763851.147693:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763851.147697:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789763851.147705:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763851.147711:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3830 00000100:00080000:0.0:1789763851.147726:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763851.147736:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763851.147748:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763851.147774:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763851/real 1789763851] req@ffff8dd211834e00 x1876699496276480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763851.147796:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496276864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763851.147807:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496276480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763851.147852:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763851.147859:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763851.147865:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496276480:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763851.147899:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496276608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763851.147927:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763851.147943:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763851/real 1789763851] req@ffff8dd211835180 x1876699496276608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763851.147958:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496276608/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763851.147969:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763851.147973:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763851.147978:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496276608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763851.147986:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496276736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763851.148007:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763851.148019:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763851/real 1789763851] req@ffff8dd211837480 x1876699496276736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763851.148033:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496276736/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763851.148047:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763851.148053:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763851.148059:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496276736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763851.148071:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496276864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763851.148107:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763851.148132:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763851/real 1789763851] req@ffff8dd211836d80 x1876699496276864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763851.148157:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496276864/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763851.148199:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763851.148204:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763851.148209:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496276864:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763851.723320:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763852.747376:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763853.771410:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763854.795387:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763855.819401:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763856.267375:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763856.267387:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763856.267392:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763856.267405:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3835 00000100:00080000:3.0:1789763856.267418:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763856.267454:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763856.267526:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145180 x1876699496276992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763856.267569:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763856.267580:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763856.267583:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763856.267589:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3835 00000100:00080000:3.0:1789763856.267595:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763856.267600:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763856.267612:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763856.267631:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496277120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763856.267640:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496276992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763856.267650:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763856.267656:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763856.267659:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763856.267664:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3835 00000100:00080000:3.0:1789763856.267669:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763856.267672:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763856.267678:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763856.267694:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496277248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763856.267706:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763856.267711:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763856.267713:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763856.267716:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3835 00000100:00080000:3.0:1789763856.267720:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763856.267729:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763856.267729:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763856.267741:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763856.267756:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496277376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763856.267791:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763856/real 1789763856] req@ffff8dd203145180 x1876699496276992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763856.267814:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145180 x1876699496276992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763856.267827:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763856.267831:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763856.267836:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496276992:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763856.267869:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496277120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763856.267921:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763856.267962:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763856/real 1789763856] req@ffff8dd203144000 x1876699496277120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763856.267981:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496277120/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763856.267992:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763856.267996:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763856.268000:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496277120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763856.268008:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496277248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763856.268035:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763856.268047:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763856/real 1789763856] req@ffff8dd203147100 x1876699496277248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763856.268063:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496277248/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763856.268088:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763856.268094:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763856.268100:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496277248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763856.268111:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496277376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763856.268148:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763856.268168:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763856/real 1789763856] req@ffff8dd203145880 x1876699496277376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763856.268251:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496277376/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763856.268262:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763856.268268:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763856.268274:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496277376:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763856.843490:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763857.867442:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763858.891323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763859.915422:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763860.939454:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763861.387309:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763861.387321:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763861.387326:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763861.387339:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3840 00000100:00080000:3.0:1789763861.387371:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763861.387378:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1789763861.387456:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147480 x1876699496277504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763861.387503:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763861.387507:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763861.387509:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763861.387519:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3840 00000100:00080000:3.0:1789763861.387523:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763861.387527:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763861.387539:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763861.387583:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203145880 x1876699496277632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763861.387599:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763861.387600:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496277504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1789763861.387610:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763861.387616:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763861.387624:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3840 00000100:00080000:3.0:1789763861.387632:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763861.387639:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763861.387648:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763861.387669:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203147100 x1876699496277760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763861.387685:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789763861.387686:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1789763861.387693:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763861.387696:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763861.387701:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3840 00000100:00080000:3.0:1789763861.387705:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763861.387711:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1789763861.387718:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763861.387732:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd203144000 x1876699496277888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763861.387763:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763861/real 1789763861] req@ffff8dd203147480 x1876699496277504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763861.387791:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147480 x1876699496277504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763861.387807:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763861.387811:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763861.387815:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496277504:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763861.387847:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496277632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763861.387882:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763861.387903:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763861/real 1789763861] req@ffff8dd203145880 x1876699496277632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763861.387922:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203145880 x1876699496277632/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763861.387930:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763861.387934:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763861.387937:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203145880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496277632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763861.387943:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496277760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763861.387965:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763861.387976:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763861/real 1789763861] req@ffff8dd203147100 x1876699496277760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763861.387987:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203147100 x1876699496277760/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763861.387995:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763861.387998:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763861.388001:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203147100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496277760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763861.388006:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496277888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763861.388020:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763861.388028:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763861/real 1789763861] req@ffff8dd203144000 x1876699496277888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763861.388040:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd203144000 x1876699496277888/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763861.388047:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763861.388050:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763861.388052:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd203144000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496277888:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763861.963280:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763862.987407:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763864.011493:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763865.035482:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763866.059330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763866.507427:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763866.507435:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763866.507439:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763866.507448:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3846 00000100:00080000:2.0:1789763866.507474:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763866.507481:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763866.507514:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496278016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763866.507543:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763866.507546:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763866.507552:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763866.507556:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3846 00000100:00080000:2.0:1789763866.507559:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763866.507562:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763866.507570:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763866.507592:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496278144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763866.507603:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496278016:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763866.507609:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763866.507616:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763866.507618:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763866.507621:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3846 00000100:00080000:2.0:1789763866.507624:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763866.507626:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763866.507631:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763866.507641:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496278272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763866.507655:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763866.507658:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763866.507660:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763866.507663:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3846 00000100:00080000:2.0:1789763866.507667:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763866.507669:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763866.507674:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763866.507676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763866.507691:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496278400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763866.507737:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763866/real 1789763866] req@ffff8dd205267100 x1876699496278016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763866.507775:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496278016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763866.507788:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763866.507792:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763866.507796:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496278016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763866.507820:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496278144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763866.507869:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763866.507894:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763866/real 1789763866] req@ffff8dd205266680 x1876699496278144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763866.507921:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496278144/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763866.507927:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763866.507929:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763866.507931:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496278144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763866.507936:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496278272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763866.507959:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763866.507986:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763866/real 1789763866] req@ffff8dd205265500 x1876699496278272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763866.508006:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496278272/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763866.508022:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763866.508028:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763866.508035:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496278272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763866.508059:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496278400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763866.508081:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763866.508093:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763866/real 1789763866] req@ffff8dd205264a80 x1876699496278400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763866.508102:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496278400/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763866.508124:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763866.508126:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763866.508129:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496278400:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763867.083420:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763868.107414:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763869.131419:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763870.155349:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763871.179502:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763871.627373:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763871.627384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763871.627389:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763871.627401:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3851 00000100:00080000:0.0:1789763871.627415:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763871.627423:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763871.627604:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496278528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763871.627663:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763871.627670:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763871.627674:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763871.627683:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3851 00000100:00080000:0.0:1789763871.627697:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763871.627712:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763871.627712:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496278528:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789763871.627745:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763871.627779:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496278656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763871.627794:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763871.627799:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763871.627802:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763871.627809:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3851 00000100:00080000:0.0:1789763871.627813:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763871.627822:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763871.627824:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763871.627833:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763871.627860:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496278784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763871.627883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763871.627895:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789763871.627895:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763871/real 1789763871] req@ffff8dd211834a80 x1876699496278528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763871.627906:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763871.627926:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3851 00000100:00100000:1.0:1789763871.627933:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496278528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763871.627937:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763871.627955:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1789763871.627957:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789763871.627964:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763871.627965:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763871.627974:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496278528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763871.627991:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496278912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763871.628024:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496278656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763871.628052:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763871.628078:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763871/real 1789763871] req@ffff8dd211836680 x1876699496278656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763871.628088:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496278656/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763871.628095:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763871.628098:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763871.628100:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496278656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763871.628105:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496278784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763871.628120:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763871.628127:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763871/real 1789763871] req@ffff8dd211834380 x1876699496278784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763871.628135:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496278784/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763871.628141:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763871.628143:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763871.628145:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496278784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763871.628149:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496278912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763871.628161:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763871.628187:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763871/real 1789763871] req@ffff8dd211835f80 x1876699496278912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763871.628197:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496278912/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763871.628207:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763871.628210:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763871.628213:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496278912:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763872.203429:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763873.227469:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763874.251374:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763875.275444:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763876.299416:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763876.747363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763876.747384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763876.747393:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763876.747413:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3856 00000100:00080000:0.0:1789763876.747435:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763876.747448:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763876.747514:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496279040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763876.747558:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763876.747563:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763876.747566:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763876.747571:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3856 00000100:00080000:0.0:1789763876.747576:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763876.747581:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763876.747599:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763876.747633:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496279040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763876.747634:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496279168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763876.747667:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763876.747674:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763876.747678:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763876.747686:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3856 00000100:00080000:0.0:1789763876.747693:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763876.747702:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763876.747714:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763876.747739:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496279296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763876.747776:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763876.747780:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763876.747783:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763876.747793:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3856 00000100:00080000:0.0:1789763876.747801:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763876.747810:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763876.747823:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763876.747849:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496279168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1789763876.747854:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496279424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763876.747924:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496279296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763876.747989:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496279424:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763877.323451:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763878.347497:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763879.371402:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763879.819467:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1789763879.819514:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763879.819533:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763879.819575:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763879.819587:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1789763879.819599:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1789763879.819651:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763876/real 1789763879] req@ffff8dd211835c00 x1876699496279424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.819698:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496279424/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.819712:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.819718:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763879.819724:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496279424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763879.819733:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763876/real 1789763879] req@ffff8dd211834000 x1876699496279168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.819746:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496279168/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.819755:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.819761:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763879.819765:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496279168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763879.819774:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763876/real 1789763879] req@ffff8dd211836a00 x1876699496279296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.819788:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496279296/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.819797:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.819802:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763879.819806:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496279296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763879.819814:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763876/real 1789763879] req@ffff8dd211834700 x1876699496279040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.819829:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496279040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.819839:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.819843:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763879.819847:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496279040:192.168.203.109@tcp:250:kworker.0 00010000:00010000:0.0:1789763880.395461:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763881.419432:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763881.867354:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763881.867364:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763881.867368:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763881.867399:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3861 00000100:00080000:0.0:1789763881.867428:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763881.867449:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763881.867507:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496279552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763881.867558:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763881.867568:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763881.867573:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763881.867620:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3861 00000100:00080000:0.0:1789763881.867625:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763881.867633:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763881.867635:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496279552:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789763881.867645:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763881.867665:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496279680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763881.867687:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763881.867691:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763881.867693:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763881.867697:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3861 00000100:00080000:0.0:1789763881.867709:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763881.867713:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763881.867721:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1789763881.867723:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763881.867738:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496279808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763881.867758:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763881.867762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763881.867764:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763881.867768:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3861 00000100:00080000:0.0:1789763881.867771:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763881.867777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763881/real 1789763881] req@ffff8dd211837800 x1876699496279552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763881.867778:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763881.867787:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763881.867802:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496279936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763881.867815:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496279552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763881.867832:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763881.867836:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763881.867840:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496279552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763881.867880:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496279680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763881.867915:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763881.867935:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763881/real 1789763881] req@ffff8dd211837b80 x1876699496279680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763881.867955:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496279680/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763881.867963:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763881.867969:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763881.867974:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496279680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763881.867983:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496279808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763881.868016:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763881.868033:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763881/real 1789763881] req@ffff8dd211837100 x1876699496279808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763881.868051:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496279808/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763881.868064:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763881.868068:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763881.868073:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496279808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763881.868083:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496279936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763881.868111:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763881.868136:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763881/real 1789763881] req@ffff8dd211836300 x1876699496279936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763881.868154:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496279936/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763881.868189:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763881.868195:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763881.868200:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496279936:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763882.443406:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763883.467411:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763884.491421:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763885.515466:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763886.539420:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763886.987352:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763886.987380:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763886.987388:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763886.987405:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3866 00000100:00080000:2.0:1789763886.987422:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763886.987435:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763886.987484:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267480 x1876699496280064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763886.987532:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763886.987537:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763886.987541:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763886.987548:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3866 00000100:00080000:2.0:1789763886.987554:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763886.987561:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763886.987574:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763886.987593:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496280064:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763886.987612:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496280192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763886.987630:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763886.987635:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763886.987639:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763886.987644:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3866 00000100:00080000:2.0:1789763886.987652:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763886.987657:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763886.987667:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763886.987682:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763886.987696:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496280320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763886.987720:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763886.987728:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763886.987737:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763886.987738:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763886/real 1789763886] req@ffff8dd205267480 x1876699496280064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763886.987760:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3866 00000100:00080000:2.0:1789763886.987775:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763886.987783:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267480 x1876699496280064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763886.987792:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763886.987812:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763886.987816:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763886.987826:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763886.987834:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496280064:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763886.987843:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496280448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763886.987877:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496280192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763886.987911:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763886.987927:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763886/real 1789763886] req@ffff8dd205264a80 x1876699496280192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763886.987945:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496280192/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763886.987956:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763886.987960:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763886.987964:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496280192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763886.987971:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496280320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763886.987991:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763886.988003:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763886/real 1789763886] req@ffff8dd205265500 x1876699496280320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763886.988017:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496280320/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763886.988027:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763886.988039:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763886.988043:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496280320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763886.988049:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496280448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763886.988068:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763886.988079:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763886/real 1789763886] req@ffff8dd205266680 x1876699496280448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763886.988093:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496280448/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763886.988111:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763886.988116:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763886.988119:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496280448:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763887.563412:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763888.587480:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763889.611391:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763890.635353:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763891.659352:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763892.107286:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763892.107297:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763892.107302:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763892.107313:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3871 00000100:00080000:0.0:1789763892.107324:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763892.107331:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763892.107397:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836300 x1876699496280576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763892.107433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763892.107438:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763892.107441:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763892.107446:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3871 00000100:00080000:0.0:1789763892.107452:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763892.107458:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763892.107471:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763892.107504:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496280576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763892.107507:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837100 x1876699496280704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763892.107520:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763892.107534:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763892.107536:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763892.107541:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3871 00000100:00080000:0.0:1789763892.107546:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763892.107552:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763892.107558:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763892.107570:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837b80 x1876699496280832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763892.107581:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763892.107585:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763892.107589:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763892.107591:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763892.107598:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3871 00000100:00080000:0.0:1789763892.107604:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763892.107609:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763892.107614:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763892.107628:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837800 x1876699496280960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763892.107645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763892/real 1789763892] req@ffff8dd211836300 x1876699496280576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.107668:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836300 x1876699496280576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.107680:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.107685:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763892.107689:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496280576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763892.107713:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496280704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763892.107734:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763892.107746:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763892/real 1789763892] req@ffff8dd211837100 x1876699496280704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.107758:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837100 x1876699496280704/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.107766:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.107769:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763892.107772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496280704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763892.107778:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496280832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763892.107795:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763892.107816:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763892/real 1789763892] req@ffff8dd211837b80 x1876699496280832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.107828:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837b80 x1876699496280832/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.107836:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.107839:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763892.107842:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496280832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763892.107866:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496280960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763892.107902:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763892.107925:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763892/real 1789763892] req@ffff8dd211837800 x1876699496280960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763892.107949:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837800 x1876699496280960/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763892.107965:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763892.107972:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763892.107978:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496280960:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763892.683304:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763893.707376:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763894.731368:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763895.755401:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763896.779379:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763897.227275:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763897.227288:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763897.227293:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763897.227306:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3876 00000100:00080000:0.0:1789763897.227319:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763897.227328:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763897.227375:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834700 x1876699496281088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763897.227421:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763897.227426:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763897.227429:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763897.227434:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3876 00000100:00080000:0.0:1789763897.227438:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763897.227442:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763897.227454:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763897.227469:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836a00 x1876699496281216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763897.227483:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763897.227488:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763897.227491:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763897.227499:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3876 00000100:00080000:0.0:1789763897.227505:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763897.227514:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763897.227526:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496281088:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1789763897.227527:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763897.227578:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834000 x1876699496281344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763897.227603:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763897.227608:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763897.227611:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763897.227617:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3876 00000100:00080000:0.0:1789763897.227627:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763897.227633:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763897.227643:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763897.227643:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763897.227669:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835c00 x1876699496281472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763897.227708:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763897/real 1789763897] req@ffff8dd211834700 x1876699496281088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763897.227734:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834700 x1876699496281088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763897.227748:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763897.227753:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763897.227759:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496281088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763897.227769:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496281216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763897.227796:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763897.227811:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763897/real 1789763897] req@ffff8dd211836a00 x1876699496281216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763897.227826:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836a00 x1876699496281216/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763897.227837:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763897.227841:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763897.227845:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496281216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763897.227875:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496281344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763897.227897:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763897.227908:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763897/real 1789763897] req@ffff8dd211834000 x1876699496281344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763897.227923:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834000 x1876699496281344/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763897.227941:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763897.227945:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763897.227949:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496281344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763897.227956:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496281472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763897.227974:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763897.227985:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763897/real 1789763897] req@ffff8dd211835c00 x1876699496281472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763897.227998:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835c00 x1876699496281472/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763897.228008:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763897.228012:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763897.228016:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496281472:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763897.803379:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763898.827357:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763899.851436:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763900.875322:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763901.899432:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763902.347374:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763902.347391:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763902.347398:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763902.347416:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3881 00000100:00080000:0.0:1789763902.347434:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763902.347446:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763902.347520:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496281600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763902.347562:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763902.347569:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763902.347573:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763902.347581:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3881 00000100:00080000:0.0:1789763902.347591:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763902.347601:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763902.347620:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763902.347632:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496281600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763902.347658:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496281728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763902.347683:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763902.347688:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763902.347690:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763902.347696:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3881 00000100:00080000:0.0:1789763902.347705:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763902.347710:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763902.347720:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763902.347721:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763902.347752:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496281856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763902.347767:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763902.347783:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763902.347786:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763902.347793:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3881 00000100:00080000:0.0:1789763902.347802:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763902.347811:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763902.347820:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763902.347833:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763902/real 1789763902] req@ffff8dd211835f80 x1876699496281600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763902.347836:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496281984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763902.347931:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496281600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763902.347964:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763902.347971:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763902.347976:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496281600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763902.348014:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496281728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763902.348054:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763902.348073:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763902/real 1789763902] req@ffff8dd211834380 x1876699496281728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763902.348136:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496281728/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763902.348146:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763902.348149:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763902.348152:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496281728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763902.348159:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496281856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763902.348241:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763902.348258:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763902/real 1789763902] req@ffff8dd211836680 x1876699496281856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763902.348274:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496281856/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763902.348285:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763902.348289:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763902.348293:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496281856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763902.348301:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496281984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763902.348324:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763902.348336:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763902/real 1789763902] req@ffff8dd211834a80 x1876699496281984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763902.348352:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496281984/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763902.348362:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763902.348365:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763902.348369:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496281984:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763902.923433:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763903.947383:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763904.971407:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763905.995363:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763907.019285:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763907.467250:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763907.467258:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763907.467261:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763907.467269:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3887 00000100:00080000:2.0:1789763907.467279:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763907.467286:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763907.467333:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496282112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763907.467355:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763907.467357:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763907.467359:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763907.467362:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3887 00000100:00080000:2.0:1789763907.467365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763907.467367:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763907.467376:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763907.467387:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496282240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763907.467395:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763907.467398:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763907.467399:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496282112:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763907.467400:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763907.467407:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3887 00000100:00080000:2.0:1789763907.467411:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763907.467414:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763907.467418:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763907.467426:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496282368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763907.467433:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763907.467436:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763907.467437:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763907.467441:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3887 00000100:00080000:2.0:1789763907.467443:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763907.467446:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763907.467450:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763907.467458:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496282496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763907.467470:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763907.467512:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763907/real 1789763907] req@ffff8dd205267100 x1876699496282112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763907.467532:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496282112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763907.467543:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763907.467547:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763907.467552:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496282112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763907.467575:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496282240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763907.467596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763907.467608:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763907/real 1789763907] req@ffff8dd205264380 x1876699496282240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763907.467621:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496282240/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763907.467637:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763907.467641:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763907.467644:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496282240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763907.467650:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496282368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763907.467666:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763907.467675:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763907/real 1789763907] req@ffff8dd205264e00 x1876699496282368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763907.467687:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496282368/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763907.467696:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763907.467699:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763907.467703:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496282368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763907.467709:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496282496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763907.467727:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763907.467738:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763907/real 1789763907] req@ffff8dd205267800 x1876699496282496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763907.467750:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496282496/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763907.467759:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763907.467762:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763907.467766:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496282496:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763908.043391:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763909.067435:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763910.091398:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763911.115430:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763912.139376:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763912.587325:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763912.587346:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763912.587355:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763912.587377:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3892 00000100:00080000:2.0:1789763912.587401:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763912.587416:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763912.587493:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496282624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763912.587552:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763912.587560:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763912.587565:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763912.587573:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3892 00000100:00080000:2.0:1789763912.587580:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763912.587587:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763912.587604:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763912.587616:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496282624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763912.587631:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496282752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763912.587650:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763912.587655:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763912.587658:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763912.587663:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3892 00000100:00080000:2.0:1789763912.587668:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763912.587674:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763912.587681:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763912.587696:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496282880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763912.587709:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763912.587714:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763912.587717:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763912.587721:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3892 00000100:00080000:2.0:1789763912.587725:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763912.587729:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763912.587735:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763912.587746:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763912.587752:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496283008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763912.587854:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763912/real 1789763912] req@ffff8dd205264000 x1876699496282624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763912.587885:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496282624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763912.587901:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763912.587907:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763912.587913:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496282624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763912.587952:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496282752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763912.587987:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763912.588005:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763912/real 1789763912] req@ffff8dd205265880 x1876699496282752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763912.588062:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496282752/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763912.588080:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763912.588087:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763912.588094:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496282752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763912.588108:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496282880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763912.588150:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763912.588204:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763912/real 1789763912] req@ffff8dd205267b80 x1876699496282880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763912.588222:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496282880/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763912.588233:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763912.588237:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763912.588241:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496282880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763912.588249:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496283008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763912.588277:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763912.588291:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763912/real 1789763912] req@ffff8dd205266a00 x1876699496283008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763912.588306:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496283008/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763912.588316:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763912.588319:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763912.588323:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496283008:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763913.163382:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763914.187436:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763915.211340:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763916.235420:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763917.259390:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763917.707375:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763917.707392:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763917.707400:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763917.707416:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3897 00000100:00080000:1.0:1789763917.707434:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763917.707446:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763917.707504:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496283136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763917.707548:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763917.707553:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763917.707556:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763917.707561:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3897 00000100:00080000:1.0:1789763917.707567:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763917.707573:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763917.707587:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763917.707609:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496283264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763917.707612:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496283136:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763917.707640:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763917.707644:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763917.707647:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763917.707652:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3897 00000100:00080000:1.0:1789763917.707656:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763917.707660:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763917.707667:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763917.707683:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496283392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763917.707697:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763917.707702:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763917.707705:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763917.707710:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3897 00000100:00080000:1.0:1789763917.707716:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763917.707720:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763917.707727:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763917.707734:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763917.707746:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496283520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763917.707811:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763917/real 1789763917] req@ffff8dd20504ea00 x1876699496283136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763917.707838:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496283136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763917.707853:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763917.707858:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763917.707863:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496283136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763917.707896:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496283264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763917.707928:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763917.707943:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763917/real 1789763917] req@ffff8dd20504d880 x1876699496283264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763917.707959:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496283264/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763917.707970:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763917.708001:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763917.708006:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496283264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763917.708015:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496283392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763917.708044:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763917.708059:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763917/real 1789763917] req@ffff8dd20504ed80 x1876699496283392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763917.708074:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496283392/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763917.708093:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763917.708097:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763917.708101:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496283392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763917.708108:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496283520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763917.708127:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763917.708138:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763917/real 1789763917] req@ffff8dd20504f100 x1876699496283520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763917.708152:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496283520/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763917.708162:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763917.708165:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763917.708201:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496283520:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763918.283266:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763919.307386:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763920.331284:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763921.355423:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763922.379421:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763922.827404:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763922.827416:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763922.827422:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763922.827435:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3902 00000100:00080000:0.0:1789763922.827449:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763922.827457:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763922.827505:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496283648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763922.827541:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763922.827548:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763922.827551:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763922.827560:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3902 00000100:00080000:0.0:1789763922.827567:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763922.827576:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763922.827594:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763922.827611:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496283648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1789763922.827648:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496283776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763922.827680:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763922.827685:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763922.827689:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763922.827694:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3902 00000100:00080000:0.0:1789763922.827701:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763922.827705:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763922.827713:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763922.827726:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763922.827748:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496283904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763922.827762:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763922.827769:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763922/real 1789763922] req@ffff8dd211836d80 x1876699496283648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763922.827771:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763922.827776:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763922.827790:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3902 00000100:00080000:0.0:1789763922.827800:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763922.827809:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763922.827816:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763922.827819:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496283648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763922.827837:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496284032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763922.827838:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763922.827850:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763922.827857:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496283648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763922.827901:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496283776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763922.827935:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763922.827981:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763922/real 1789763922] req@ffff8dd211837480 x1876699496283776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763922.827998:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496283776/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763922.828009:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763922.828013:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763922.828018:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496283776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763922.828025:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496283904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763922.828051:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763922.828063:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763922/real 1789763922] req@ffff8dd211835180 x1876699496283904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763922.828077:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496283904/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763922.828087:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763922.828090:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763922.828094:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496283904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763922.828101:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496284032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763922.828119:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763922.828132:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763922/real 1789763922] req@ffff8dd211834e00 x1876699496284032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763922.828146:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496284032/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763922.828155:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763922.828158:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763922.828162:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496284032:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763923.403393:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763924.427331:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763925.451387:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763926.475430:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763927.499405:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763927.947298:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763927.947314:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763927.947321:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763927.947337:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3907 00000100:00080000:0.0:1789763927.947355:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763927.947367:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763927.947428:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834e00 x1876699496284160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763927.947468:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763927.947473:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763927.947476:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763927.947481:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3907 00000100:00080000:0.0:1789763927.947487:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763927.947492:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763927.947505:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763927.947523:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835180 x1876699496284288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763927.947540:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763927.947549:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496284160:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763927.947550:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763927.947557:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763927.947572:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3907 00000100:00080000:0.0:1789763927.947576:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763927.947581:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763927.947588:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763927.947603:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211837480 x1876699496284416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763927.947618:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763927.947623:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763927.947626:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763927.947630:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3907 00000100:00080000:0.0:1789763927.947636:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763927.947640:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763927.947647:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763927.947655:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1789763927.947671:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836d80 x1876699496284544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763927.947720:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763927/real 1789763927] req@ffff8dd211834e00 x1876699496284160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763927.947746:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834e00 x1876699496284160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763927.947761:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763927.947766:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763927.947772:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496284160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763927.947800:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496284288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763927.947828:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763927.947843:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763927/real 1789763927] req@ffff8dd211835180 x1876699496284288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763927.947859:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835180 x1876699496284288/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763927.947871:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763927.947875:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763927.947879:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496284288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763927.947886:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496284416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763927.947944:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763927.947966:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763927/real 1789763927] req@ffff8dd211837480 x1876699496284416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763927.947990:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211837480 x1876699496284416/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763927.948007:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763927.948014:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763927.948021:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211837480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496284416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763927.948032:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496284544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763927.948061:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763927.948077:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763927/real 1789763927] req@ffff8dd211836d80 x1876699496284544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763927.948091:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836d80 x1876699496284544/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763927.948101:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763927.948105:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763927.948110:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496284544:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763928.523446:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763929.547413:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763930.571425:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763931.595408:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763932.619437:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763933.067320:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763933.067335:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763933.067354:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763933.067369:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3912 00000100:00080000:0.0:1789763933.067384:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763933.067395:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1789763933.067451:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834a80 x1876699496284672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763933.067493:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763933.067501:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763933.067504:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763933.067514:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3912 00000100:00080000:0.0:1789763933.067521:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763933.067529:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763933.067546:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763933.067573:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211836680 x1876699496284800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763933.067592:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763933.067599:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763933.067603:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763933.067617:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3912 00000100:00100000:2.0:1789763933.067620:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496284672:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1789763933.067644:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763933.067654:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763933.067666:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763933.067695:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211834380 x1876699496284928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763933.067713:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763933.067718:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763933.067720:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763933.067732:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3912 00000800:00000100:2.0:1789763933.067733:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1789763933.067743:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763933.067751:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1789763933.067760:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763933.067777:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd211835f80 x1876699496285056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763933.067803:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763933/real 1789763933] req@ffff8dd211834a80 x1876699496284672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763933.067833:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834a80 x1876699496284672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763933.067849:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763933.067858:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763933.067897:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496284672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763933.067915:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496284800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763933.067967:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763933.067993:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763933/real 1789763933] req@ffff8dd211836680 x1876699496284800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763933.068011:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211836680 x1876699496284800/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763933.068022:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763933.068026:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763933.068030:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211836680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496284800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1789763933.068061:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496284928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1789763933.068090:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763933.068104:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763933/real 1789763933] req@ffff8dd211834380 x1876699496284928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763933.068118:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211834380 x1876699496284928/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763933.068129:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763933.068133:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763933.068137:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211834380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496284928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1789763933.068143:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496285056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1789763933.068162:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763933.068218:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763933/real 1789763933] req@ffff8dd211835f80 x1876699496285056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763933.068234:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd211835f80 x1876699496285056/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763933.068244:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763933.068248:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1789763933.068252:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd211835f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496285056:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763933.643456:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763934.667297:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763935.691446:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763936.715411:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763937.739387:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763938.187287:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763938.187299:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763938.187304:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763938.187315:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3917 00000100:00080000:2.0:1789763938.187327:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763938.187352:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763938.187396:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496285184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763938.187436:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763938.187440:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763938.187443:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763938.187448:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3917 00000100:00080000:2.0:1789763938.187452:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763938.187456:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763938.187467:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763938.187483:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496285312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763938.187512:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763938.187516:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763938.187518:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763938.187526:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3917 00000100:00100000:1.0:1789763938.187530:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496285184:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763938.187531:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763938.187541:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763938.187549:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763938.187565:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496285440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763938.187589:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763938.187593:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763938.187595:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763938.187600:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3917 00000100:00080000:2.0:1789763938.187605:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763938.187609:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1789763938.187617:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1789763938.187619:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763938.187638:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496285568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763938.187674:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763938/real 1789763938] req@ffff8dd205264700 x1876699496285184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763938.187696:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496285184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763938.187709:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763938.187713:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763938.187717:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496285184:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763938.187725:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496285312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763938.187747:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763938.187760:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763938/real 1789763938] req@ffff8dd205265f80 x1876699496285312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763938.187772:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496285312/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763938.187781:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763938.187785:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763938.187788:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496285312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763938.187814:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496285440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763938.187859:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763938.187871:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763938/real 1789763938] req@ffff8dd205265180 x1876699496285440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763938.187883:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496285440/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763938.187892:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763938.187895:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763938.187898:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496285440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763938.187904:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496285568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763938.187921:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763938.187929:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763938/real 1789763938] req@ffff8dd205266d80 x1876699496285568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763938.187941:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496285568/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763938.187949:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763938.187952:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763938.187954:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496285568:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763938.763409:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763939.787367:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763940.811403:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763941.835465:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763942.859434:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763943.307365:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763943.307378:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763943.307384:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763943.307412:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3922 00000100:00080000:2.0:1789763943.307427:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763943.307436:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763943.307486:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266300 x1876699496285696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763943.307530:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763943.307536:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763943.307538:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763943.307543:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3922 00000100:00080000:2.0:1789763943.307547:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763943.307551:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763943.307564:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763943.307584:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266d80 x1876699496285824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763943.307599:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763943.307603:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763943.307605:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763943.307610:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3922 00000100:00080000:2.0:1789763943.307614:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763943.307619:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763943.307625:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763943.307638:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496285696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763943.307647:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265180 x1876699496285952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763943.307671:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763943.307674:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763943.307676:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763943.307682:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3922 00000100:00080000:2.0:1789763943.307686:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763943.307692:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763943.307698:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763943.307713:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265f80 x1876699496286080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763943.307869:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496285824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763943.307939:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496285952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763943.307971:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496286080:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763943.883441:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763944.907390:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763945.931400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763946.379681:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1789763946.379817:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763946.379847:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763946.379939:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763946.379959:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1789763946.379976:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1789763946.380241:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763943/real 1789763946] req@ffff8dd205265180 x1876699496285952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763946.380301:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265180 x1876699496285952/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763946.380325:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763946.380334:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763946.380365:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496285952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763946.380385:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763943/real 1789763946] req@ffff8dd205265f80 x1876699496286080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763946.380411:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265f80 x1876699496286080/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763946.380428:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763946.380435:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763946.380458:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496286080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763946.380473:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763943/real 1789763946] req@ffff8dd205266300 x1876699496285696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763946.380495:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266300 x1876699496285696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763946.380510:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763946.380516:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763946.380522:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496285696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763946.380535:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763943/real 1789763946] req@ffff8dd205266d80 x1876699496285824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763946.380557:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266d80 x1876699496285824/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763946.380586:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763946.380594:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763946.380601:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496285824:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763946.955383:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763947.979390:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763948.427321:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763948.427333:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763948.427338:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763948.427349:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3928 00000100:00080000:2.0:1789763948.427362:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763948.427408:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763948.427456:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264700 x1876699496286208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763948.427494:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763948.427500:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763948.427503:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763948.427510:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3928 00000100:00080000:2.0:1789763948.427517:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763948.427524:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763948.427538:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763948.427557:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496286208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1789763948.427565:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266a00 x1876699496286336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763948.427587:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763948.427593:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763948.427597:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763948.427604:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3928 00000100:00080000:2.0:1789763948.427612:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763948.427619:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763948.427629:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763948.427646:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763948.427654:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267b80 x1876699496286464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763948.427684:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763948.427690:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763948.427694:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763948.427702:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3928 00000100:00080000:2.0:1789763948.427709:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763948.427712:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763948/real 1789763948] req@ffff8dd205264700 x1876699496286208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763948.427721:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763948.427738:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763948.427781:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264700 x1876699496286208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763948.427787:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265880 x1876699496286592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763948.427830:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763948.427842:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763948.427851:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496286208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1789763948.427891:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496286336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763948.427944:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763948.427975:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763948/real 1789763948] req@ffff8dd205266a00 x1876699496286336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763948.427996:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266a00 x1876699496286336/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763948.428009:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763948.428014:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763948.428020:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496286336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1789763948.428031:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496286464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1789763948.428065:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763948.428086:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763948/real 1789763948] req@ffff8dd205267b80 x1876699496286464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763948.428118:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267b80 x1876699496286464/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763948.428131:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763948.428136:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763948.428142:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496286464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1789763948.428151:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496286592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1789763948.428221:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763948.428241:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763948/real 1789763948] req@ffff8dd205265880 x1876699496286592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763948.428260:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265880 x1876699496286592/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763948.428282:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763948.428287:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1789763948.428292:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496286592:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763949.003366:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763950.027405:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763951.051352:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763952.075385:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763953.099447:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763953.547236:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763953.547246:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763953.547250:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763953.547260:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3933 00000100:00080000:2.0:1789763953.547270:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763953.547276:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763953.547329:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264000 x1876699496286720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763953.547358:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763953.547361:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763953.547363:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763953.547375:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3933 00000100:00080000:2.0:1789763953.547381:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763953.547385:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763953.547395:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763953.547410:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267800 x1876699496286848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763953.547417:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496286720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763953.547427:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763953.547448:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763953.547450:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763953.547454:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3933 00000100:00080000:2.0:1789763953.547458:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763953.547461:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763953.547466:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763953.547478:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264e00 x1876699496286976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763953.547487:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763953.547491:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763953.547493:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763953.547495:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3933 00000100:00080000:2.0:1789763953.547517:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789763953.547524:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1789763953.547530:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763953.547537:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763953.547559:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264380 x1876699496287104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763953.547581:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763953/real 1789763953] req@ffff8dd205264000 x1876699496286720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763953.547632:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264000 x1876699496286720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763953.547647:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763953.547653:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763953.547661:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496286720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763953.547707:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496286848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763953.547783:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763953.547796:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763953/real 1789763953] req@ffff8dd205267800 x1876699496286848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763953.547806:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267800 x1876699496286848/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763953.547813:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763953.547816:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763953.547819:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496286848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763953.547825:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496286976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763953.547840:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763953.547848:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763953/real 1789763953] req@ffff8dd205264e00 x1876699496286976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763953.547860:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264e00 x1876699496286976/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763953.547887:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763953.547892:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763953.547897:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496286976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763953.547905:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496287104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763953.547935:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763953.547950:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763953/real 1789763953] req@ffff8dd205264380 x1876699496287104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763953.547961:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264380 x1876699496287104/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763953.547969:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763953.547972:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763953.547975:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496287104:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763954.123361:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763955.147408:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763956.171466:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763957.195380:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763958.219362:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763958.667393:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763958.667406:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763958.667411:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763958.667424:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3938 00000100:00080000:2.0:1789763958.667438:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763958.667449:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1789763958.667500:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205267100 x1876699496287232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763958.667532:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763958.667537:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763958.667540:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763958.667545:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3938 00000100:00080000:2.0:1789763958.667549:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763958.667553:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763958.667581:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763958.667597:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205266680 x1876699496287360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763958.667612:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763958.667616:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763958.667618:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763958.667626:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3938 00000100:00100000:3.0:1789763958.667628:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496287232:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1789763958.667630:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763958.667644:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763958.667652:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763958.667669:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205265500 x1876699496287488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763958.667724:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763958.667729:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763958.667732:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763958.667737:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3938 00000100:00080000:2.0:1789763958.667753:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763958.667759:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1789763958.667767:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763958.667776:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1789763958.667791:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd205264a80 x1876699496287616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763958.667838:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763958/real 1789763958] req@ffff8dd205267100 x1876699496287232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763958.667867:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205267100 x1876699496287232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763958.667881:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763958.667904:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763958.667909:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205267100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496287232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763958.667937:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496287360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763958.667992:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763958.668009:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763958/real 1789763958] req@ffff8dd205266680 x1876699496287360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763958.668025:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205266680 x1876699496287360/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763958.668036:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763958.668040:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763958.668044:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205266680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496287360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763958.668051:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496287488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763958.668072:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763958.668084:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763958/real 1789763958] req@ffff8dd205265500 x1876699496287488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763958.668098:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205265500 x1876699496287488/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763958.668116:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763958.668120:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763958.668124:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205265500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496287488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763958.668130:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496287616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763958.668149:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763958.668160:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763958/real 1789763958] req@ffff8dd205264a80 x1876699496287616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763958.668173:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd205264a80 x1876699496287616/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763958.668201:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763958.668204:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763958.668207:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd205264a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496287616:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763959.243317:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763960.267289:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763961.292296:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763962.315446:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763963.339402:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763963.787312:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763963.787328:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763963.787334:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763963.787346:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3943 00000100:00080000:1.0:1789763963.787360:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763963.787368:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763963.787416:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496287744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763963.787455:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763963.787460:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763963.787463:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763963.787468:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3943 00000100:00080000:1.0:1789763963.787472:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763963.787476:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763963.787490:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763963.787505:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496287872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763963.787521:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763963.787527:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496287744:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763963.787528:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763963.787535:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763963.787565:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3943 00000100:00080000:1.0:1789763963.787571:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763963.787576:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763963.787584:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763963.787602:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496288000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763963.787623:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763963.787632:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1789763963.787633:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1789763963.787661:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763963.787674:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3943 00000100:00080000:1.0:1789763963.787685:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763963.787692:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763963.787702:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763963.787722:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496288128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763963.787739:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763963/real 1789763963] req@ffff8dd20504c700 x1876699496287744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763963.787772:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496287744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763963.787786:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763963.787792:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763963.787797:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496287744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763963.787828:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496287872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763963.787861:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763963.787877:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763963/real 1789763963] req@ffff8dd20504f800 x1876699496287872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763963.787893:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496287872/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763963.787904:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763963.787909:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763963.787913:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496287872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763963.787921:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496288000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763963.787941:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763963.787953:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763963/real 1789763963] req@ffff8dd20504fb80 x1876699496288000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763963.787967:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496288000/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763963.787978:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763963.787982:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763963.787986:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496288000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763963.787992:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496288128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763963.788011:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763963.788023:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763963/real 1789763963] req@ffff8dd20504ce00 x1876699496288128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763963.788038:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496288128/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763963.788047:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763963.788051:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763963.788055:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496288128:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763964.363330:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763965.387367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763966.411400:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763967.435316:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763968.459390:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763968.907269:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763968.907280:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763968.907285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763968.907311:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3948 00000100:00080000:1.0:1789763968.907321:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763968.907326:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763968.907357:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496288256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763968.907413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763968.907418:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763968.907420:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763968.907428:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3948 00000100:00080000:1.0:1789763968.907432:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763968.907436:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763968.907447:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763968.907461:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496288384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763968.907471:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763968.907476:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763968.907478:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763968.907482:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3948 00000100:00080000:1.0:1789763968.907486:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763968.907490:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763968.907495:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763968.907505:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496288512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763968.907516:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763968.907522:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763968.907526:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789763968.907527:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496288256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763968.907533:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3948 00000100:00080000:1.0:1789763968.907543:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763968.907547:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763968.907552:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763968.907565:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496288640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763968.907647:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763968.907682:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763968/real 1789763968] req@ffff8dd20504d180 x1876699496288256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763968.907715:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496288256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763968.907726:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763968.907730:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763968.907735:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496288256:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763968.907743:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496288384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763968.907768:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763968.907780:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763968/real 1789763968] req@ffff8dd20504e300 x1876699496288384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763968.907792:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496288384/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763968.907808:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763968.907811:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763968.907814:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496288384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763968.907840:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496288512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763968.907857:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763968.907866:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763968/real 1789763968] req@ffff8dd20504d500 x1876699496288512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763968.907878:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496288512/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763968.907886:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763968.907889:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763968.907892:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496288512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763968.907898:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496288640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763968.907912:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763968.907921:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763968/real 1789763968] req@ffff8dd20504df80 x1876699496288640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763968.907933:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496288640/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763968.907940:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763968.907943:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763968.907947:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496288640:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763969.483351:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763970.507332:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763971.531318:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763972.555310:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763973.579318:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763974.027268:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763974.027279:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763974.027282:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763974.027303:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3953 00000100:00080000:1.0:1789763974.027314:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763974.027321:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763974.027356:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e680 x1876699496288768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763974.027381:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763974.027384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763974.027386:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763974.027389:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3953 00000100:00080000:1.0:1789763974.027391:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763974.027394:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763974.027401:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763974.027430:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504df80 x1876699496288896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763974.027438:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763974.027440:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763974.027442:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763974.027444:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3953 00000100:00080000:1.0:1789763974.027447:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763974.027451:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496288768:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763974.027453:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763974.027461:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763974.027471:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d500 x1876699496289024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763974.027497:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763974.027506:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763974.027508:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763974.027512:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3953 00000100:00080000:1.0:1789763974.027516:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763974.027520:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763974.027524:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763974.027533:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763974.027537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504e300 x1876699496289152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763974.027594:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763974/real 1789763974] req@ffff8dd20504e680 x1876699496288768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763974.027622:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e680 x1876699496288768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763974.027630:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763974.027633:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763974.027636:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496288768:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763974.027641:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496288896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763974.027660:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763974.027669:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763974/real 1789763974] req@ffff8dd20504df80 x1876699496288896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763974.027677:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504df80 x1876699496288896/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763974.027683:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763974.027685:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763974.027687:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496288896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763974.027707:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496289024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763974.027719:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763974.027727:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763974/real 1789763974] req@ffff8dd20504d500 x1876699496289024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763974.027735:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d500 x1876699496289024/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763974.027741:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763974.027743:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763974.027745:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496289024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763974.027748:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496289152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763974.027759:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763974.027765:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763974/real 1789763974] req@ffff8dd20504e300 x1876699496289152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763974.027773:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504e300 x1876699496289152/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763974.027778:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763974.027780:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763974.027782:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496289152:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763974.603268:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763975.627505:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763976.651319:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763977.675438:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763978.699339:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763979.147275:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763979.147283:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763979.147286:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763979.147295:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3958 00000100:00080000:1.0:1789763979.147304:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763979.147309:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763979.147340:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d180 x1876699496289280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763979.147360:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763979.147363:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763979.147364:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763979.147367:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3958 00000100:00080000:1.0:1789763979.147370:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763979.147372:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763979.147380:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763979.147389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ce00 x1876699496289408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763979.147396:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496289280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763979.147400:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763979.147403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763979.147405:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763979.147408:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3958 00000100:00080000:1.0:1789763979.147409:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763979.147412:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763979.147416:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763979.147425:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504fb80 x1876699496289536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763979.147432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763979.147434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763979.147435:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763979.147437:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3958 00000100:00080000:1.0:1789763979.147440:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763979.147443:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763979.147446:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763979.147453:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1789763979.147457:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f800 x1876699496289664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763979.147500:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763979/real 1789763979] req@ffff8dd20504d180 x1876699496289280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763979.147520:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d180 x1876699496289280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763979.147546:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763979.147549:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763979.147552:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496289280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763979.147572:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496289408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763979.147596:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763979.147606:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763979/real 1789763979] req@ffff8dd20504ce00 x1876699496289408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763979.147614:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ce00 x1876699496289408/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763979.147620:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763979.147622:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763979.147624:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496289408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763979.147628:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496289536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763979.147640:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763979.147645:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763979/real 1789763979] req@ffff8dd20504fb80 x1876699496289536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763979.147653:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504fb80 x1876699496289536/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763979.147658:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763979.147660:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763979.147662:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496289536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763979.147666:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496289664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763979.147676:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763979.147681:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763979/real 1789763979] req@ffff8dd20504f800 x1876699496289664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763979.147689:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f800 x1876699496289664/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763979.147693:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763979.147695:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763979.147697:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496289664:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763979.723324:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763980.747332:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763981.771316:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763982.795270:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763983.819275:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763984.267248:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763984.267259:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763984.267263:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763984.267272:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3963 00000100:00080000:1.0:1789763984.267282:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763984.267287:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763984.267322:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c700 x1876699496289792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763984.267350:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763984.267354:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763984.267356:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763984.267361:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3963 00000100:00080000:1.0:1789763984.267365:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763984.267370:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763984.267378:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763984.267389:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504f100 x1876699496289920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763984.267398:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496289792:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763984.267401:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763984.267407:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763984.267410:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763984.267413:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3963 00000100:00080000:1.0:1789763984.267416:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763984.267418:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763984.267421:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763984.267430:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ed80 x1876699496290048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763984.267438:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763984.267441:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763984.267442:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763984.267444:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3963 00000100:00080000:1.0:1789763984.267447:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763984.267450:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763984.267453:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763984.267461:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504d880 x1876699496290176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763984.267473:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763984.267529:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763984/real 1789763984] req@ffff8dd20504c700 x1876699496289792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763984.267550:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c700 x1876699496289792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763984.267561:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763984.267565:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763984.267570:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496289792:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763984.267594:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496289920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763984.267620:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763984.267634:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763984/real 1789763984] req@ffff8dd20504f100 x1876699496289920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763984.267649:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504f100 x1876699496289920/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763984.267659:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763984.267663:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763984.267667:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496289920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763984.267673:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496290048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763984.267696:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763984.267709:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763984/real 1789763984] req@ffff8dd20504ed80 x1876699496290048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763984.267722:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ed80 x1876699496290048/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763984.267731:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763984.267734:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763984.267738:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496290048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763984.267745:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496290176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763984.267766:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763984.267777:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763984/real 1789763984] req@ffff8dd20504d880 x1876699496290176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763984.267790:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504d880 x1876699496290176/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763984.267806:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763984.267809:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763984.267813:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496290176:192.168.203.109@tcp:38:kworker.0 00010000:00010000:0.0:1789763984.843305:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763985.867325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763986.891283:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763987.915323:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763988.939350:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763989.387280:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 5292164f-12cc-4fea-b062-6f9c12f1be43->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763989.387289:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763989.387292:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763989.387301:0:468:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3968 00000100:00080000:1.0:1789763989.387310:0:468:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763989.387316:0:468:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff8dd2051ff000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1789763989.387347:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ea00 x1876699496290304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763989.387369:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763989.387372:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763989.387374:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763989.387376:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3968 00000100:00080000:1.0:1789763989.387379:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763989.387381:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd2055fc800: import ffff8dd20509b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763989.387388:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763989.387398:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c380 x1876699496290432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763989.387405:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) 17022965-1e4f-46fb-915c-61730096b8e2->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763989.387408:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763989.387409:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763989.387412:0:468:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: connect to NID 192.168.203.109@tcp last attempt 3968 00000100:00080000:1.0:1789763989.387414:0:468:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763989.387419:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496290304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1789763989.387420:0:468:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dd2055fc800: import ffff8dd20509e000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763989.387427:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763989.387435:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504c000 x1876699496290560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763989.387442:0:468:0:(pinger.c:229:ptlrpc_pinger_process_import()) d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763989.387445:0:468:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763989.387446:0:468:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763989.387466:0:468:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: connect to NID 192.168.203.109@tcp last attempt 3968 00000100:00080000:1.0:1789763989.387472:0:468:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763989.387475:0:468:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dd203049000: import ffff8dd20313b000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1789763989.387479:0:468:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763989.387491:0:468:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dd20504ca80 x1876699496290688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763989.387493:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763989.387533:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763989/real 1789763989] req@ffff8dd20504ea00 x1876699496290304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763989.387554:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ea00 x1876699496290304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763989.387562:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd2051ff000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763989.387565:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763989.387568:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5292164f-12cc-4fea-b062-6f9c12f1be43:2355:1876699496290304:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1789763989.387573:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496290432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763989.387589:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763989.387597:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763989/real 1789763989] req@ffff8dd20504c380 x1876699496290432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763989.387604:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c380 x1876699496290432/t0(0) o38->lustre-MDT0000-mdc-ffff8dd2055fc800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763989.387610:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763989.387612:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763989.387614:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496290432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1789763989.387633:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496290560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1789763989.387645:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763989.387651:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763989/real 1789763989] req@ffff8dd20504c000 x1876699496290560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763989.387665:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504c000 x1876699496290560/t0(0) o8->lustre-OST0000-osc-ffff8dd2055fc800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763989.387671:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20509e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763989.387673:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763989.387675:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:17022965-1e4f-46fb-915c-61730096b8e2:2355:1876699496290560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1789763989.387679:0:2355:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496290688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1789763989.387690:0:2355:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1789763989.387696:0:2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763989/real 1789763989] req@ffff8dd20504ca80 x1876699496290688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763989.387703:0:2355:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dd20504ca80 x1876699496290688/t0(0) o38->lustre-MDT0000-mdc-ffff8dd203049000@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763989.387708:0:2355:0:(import.c:65:import_set_state_nolock()) ffff8dd20313b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763989.387710:0:2355:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1789763989.387712:0:2355:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dd20504ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d3ad1ae7-10c4-4c94-ab44-aee2b86ddea9:2355:1876699496290688:192.168.203.109@tcp:38:kworker.0 00010000:00010000:2.0:1789763989.963307:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763990.987301:0:11690:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763992.011296:0:11696:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763993.035383:0:5596:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763994.059383:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 98348 lines, 98348 kept, 0 dropped, 0 bad.